Facebook
From Ample Gibbon, 3 Years ago, written in Plain Text.
Embed
Download Paste or View Raw
Hits: 101
  1. [2020-09-29T11:53:49,593][INFO ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [40s] publication of cluster state version [442360] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  2. [2020-09-29T11:55:09,866][WARN ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [2m] publication of cluster state version [442360] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  3. [2020-09-29T11:55:49,999][INFO ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [40s] publication of cluster state version [442361] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  4. [2020-09-29T11:57:10,324][WARN ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [2m] publication of cluster state version [442361] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  5. [2020-09-29T11:57:10,324][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.08.19-12][0] marking unavailable shards as stale: [Y7ahFX6YTayeTN8XHmZzrA]
  6. [2020-09-29T11:57:10,324][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.08.21_11][0] marking unavailable shards as stale: [PXKG0icOSWq35s3svkGH9g]
  7. [2020-09-29T11:57:10,324][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.08.20-05][0] marking unavailable shards as stale: [ybW92sCMTTicXc3J357ZTg]
  8. [2020-09-29T11:57:10,324][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.08.18_09][0] marking unavailable shards as stale: [QWQkCOgPQbuQLQI6eTAPXg]
  9. [2020-09-29T11:57:10,324][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.08.18-10][0] marking unavailable shards as stale: [1Aqb17NYS5KocFawXZVf5Q]
  10. [2020-09-29T11:57:10,325][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-performance-2020.08.19][0] marking unavailable shards as stale: [ni3EgEMIR8WV8lom2J9UiQ]
  11. [2020-09-29T11:57:10,325][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-error-2020.08.19][0] marking unavailable shards as stale: [emTzUbrJR6mhv4Nyht8AhQ]
  12. [2020-09-29T11:57:10,325][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.08.19_12][0] marking unavailable shards as stale: [RIR2atT3TVqZLYHysHZGSw]
  13. [2020-09-29T11:57:10,325][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_rrs-integration-2020.08.19][0] marking unavailable shards as stale: [hRuID43RSq6-14x5sGqylQ]
  14. [2020-09-29T11:57:50,833][INFO ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [39.8s] publication of cluster state version [442362] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  15. [2020-09-29T11:59:11,131][WARN ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [2m] publication of cluster state version [442362] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  16. [2020-09-29T11:59:51,249][INFO ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [40s] publication of cluster state version [442363] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  17. [2020-09-29T12:00:09,866][WARN ][o.e.c.c.LagDetector      ] [serverra2.sit.comp.state] node [{serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true}] is lagging at cluster state version [442359], although publication of cluster state version [442360] completed [5m] ago
  18. [2020-09-29T12:01:11,509][WARN ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [2m] publication of cluster state version [442363] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  19. [2020-09-29T12:01:12,020][INFO ][o.e.c.s.MasterService    ] [serverra2.sit.comp.state] node-left[{serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} reason: lagging], term: 68, version: 442364, delta: removed {{serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true}}
  20. [2020-09-29T12:01:18,051][INFO ][o.e.c.s.ClusterApplierService] [serverra2.sit.comp.state] removed {{serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true}}, term: 68, version: 442364, reason: Publication{term=68, version=442364}
  21. [2020-09-29T12:01:18,318][INFO ][o.e.c.r.DelayedAllocationService] [serverra2.sit.comp.state] scheduling reroute for delayed shards in [29.8m] (1781 delayed shards)
  22. [2020-09-29T12:01:18,366][INFO ][o.e.c.m.MetaDataCreateIndexService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.09.29_10] creating index, cause [auto(bulk api)], templates [compp_srs], shards [1]/[2], mappings [_doc]
  23. [2020-09-29T12:01:19,260][INFO ][o.e.c.m.MetaDataCreateIndexService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.09.29-10] creating index, cause [auto(bulk api)], templates [compp_smap], shards [1]/[2], mappings [_doc]
  24. [2020-09-29T12:01:20,833][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_rrs-webservices-2020.08.18][0] marking unavailable shards as stale: [4qhwMUqCRPiMImT3bmQPRA]
  25. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.08.18-06][0] marking unavailable shards as stale: [HkZr_BrnRIGu4LnuRvjUsA]
  26. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-dao-2020.08.18][0] marking unavailable shards as stale: [D-KwSNhJT9mO45gKJHDxog]
  27. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.08.17_04][0] marking unavailable shards as stale: [_xkIG954StW1D-BHTJJfTA]
  28. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-startup-2020.08.18][0] marking unavailable shards as stale: [m114EJonSAKFhQYrxpI_sA]
  29. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.08.17-08][0] marking unavailable shards as stale: [hg1GW-KGSvW32P2d_UAEFA]
  30. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_rrs-performance-2020.08.18][0] marking unavailable shards as stale: [M6-gcM98SRm9cIMZBPPH8w]
  31. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-payment-2020.08.18][0] marking unavailable shards as stale: [cVHNrl7VRQWKnW6KTsAzgw]
  32. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.08.18-07][0] marking unavailable shards as stale: [Znz8ZdEJQK-W53JfOX1nDQ]
  33. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_web-integration-soap-2020.08.18][0] marking unavailable shards as stale: [kuxXH7DJSrSeZpzj-sN10w]
  34. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.08.18_03][0] marking unavailable shards as stale: [nGD1rud2SC20ntkgoZ7oDA]
  35. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.08.18_07][0] marking unavailable shards as stale: [oXw7h-25RQegxe1jp_rLmQ]
  36. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_web-dao-2020.08.18][0] marking unavailable shards as stale: [Y8ukfEm2Rn2s9CDDWRlD1g]
  37. [2020-09-29T12:01:20,834][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.08.21-11][0] marking unavailable shards as stale: [F6UIc2-wR3-u6U8thGy6WA]
  38. [2020-09-29T12:01:21,541][INFO ][o.e.c.m.MetaDataMappingService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.09.29-10/A3AlsvmUSjq6Hg4pW9XAxg] update_mapping [_doc]
  39. [2020-09-29T12:01:21,579][INFO ][o.e.c.m.MetaDataMappingService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.09.29_10/ZWXsImrIQXityqfEIoj31A] update_mapping [_doc]
  40. [2020-09-29T12:01:25,091][INFO ][o.e.c.m.MetaDataMappingService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.09.29-10/A3AlsvmUSjq6Hg4pW9XAxg] update_mapping [_doc]
  41. [2020-09-29T12:01:26,086][INFO ][o.e.c.m.MetaDataMappingService] [serverra2.sit.comp.state] [comp_app_compp_srs-integration-soap-2020.09.29_10/ZWXsImrIQXityqfEIoj31A] update_mapping [_doc]
  42. [2020-09-29T12:01:26,669][INFO ][o.e.x.i.IndexLifecycleTransition] [serverra2.sit.comp.state] moving index [comp_app_compp_srs-integration-soap-2020.09.29_10] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [HOT-WARM-compp]
  43. [2020-09-29T12:01:27,168][INFO ][o.e.x.i.IndexLifecycleTransition] [serverra2.sit.comp.state] moving index [comp_app_compp_smap-root-2020.09.29-10] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [HOT-WARM-compp]
  44. [2020-09-29T12:01:27,668][INFO ][o.e.x.i.IndexLifecycleTransition] [serverra2.sit.comp.state] moving index [comp_app_compp_srs-integration-soap-2020.09.29_10] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"set_priority","name":"set_priority"}] in policy [HOT-WARM-compp]
  45. [2020-09-29T12:01:29,077][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_srs-startup-2020.08.25][0] marking unavailable shards as stale: [c_P1EQEBSC2WkLhn9cIocA]
  46. [2020-09-29T12:01:29,077][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_web-dao-2020.08.25][0] marking unavailable shards as stale: [vErpIWI4S1Cqeh-wnejtjw]
  47. [2020-09-29T12:01:30,020][WARN ][o.e.c.r.a.AllocationService] [serverra2.sit.comp.state] [comp_app_compp_smap-root-2020.08.17-02][0] marking unavailable shards as stale: [8A1EF8_PRCu8GTpQyPOTZA]
  48. [2020-09-29T12:01:31,324][INFO ][o.e.c.m.MetaDataMappingService] [serverra2.sit.comp.state] [comp_app_compp_srs-audit-2020.09.29/Qx__nY4wSMa__WicWS_TuQ] update_mapping [_doc]
  49. [2020-09-29T12:01:31,365][INFO ][o.e.c.m.MetaDataMappingService] [serverra2.sit.comp.state] [comp_app_compp_rrs-daolog-2020.09.29/bLbjJFi0QDOcOsxEA2lCPw] update_mapping [_doc]
  50. [2020-09-29T12:01:38,951][INFO ][o.e.c.s.MasterService    ] [serverra2.sit.comp.state] node-join[{serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} join existing leader], term: 68, version: 442379, delta: added {{serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true}}
  51. [2020-09-29T12:02:19,129][INFO ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [40s] publication of cluster state version [442379] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  52. [2020-09-29T12:03:39,129][INFO ][o.e.c.s.ClusterApplierService] [serverra2.sit.comp.state] added {{serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true}}, term: 68, version: 442379, reason: Publication{term=68, version=442379}
  53. [2020-09-29T12:03:39,391][WARN ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [2m] publication of cluster state version [442379] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  54.  
  55. [2020-09-29T12:04:19,531][INFO ][o.e.c.c.C.CoordinatorPublication] [serverra2.sit.comp.state] after [40s] publication of cluster state version [442380] is still waiting for {serverra2_warm.sit.comp.state}{KiQQGwdoTgWwHYzMItyhXQ}{OjCMtVthSlqJk-TmL5UH8A}{10.100.24.231}{10.100.24.231:9301}{dlrt}{rack_id=rack_one, ml.machine_memory=269645852672, ml.max_open_jobs=20, xpack.installed=true, data=warm, transform.node=true} [SENT_APPLY_COMMIT]
  56.