добрый день
одна из шести нод просто “вышла” из кластера, много букв и сам я не смог разобраться
посоветуйте куда вообще смотреть в таких ситуациях и как правильно их лечить
[Tue Oct 17 09:59:37.899 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,009ab359,38)) install timer expired
[Tue Oct 17 09:59:51.416 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,3f9be3c5,41)) install timer expired
[Tue Oct 17 09:59:58.921 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,3f9be3c5,41)) install timer expired
[Tue Oct 17 10:00:07.941 2023] [713] WARNING: Quorum: No node with complete state:
[Tue Oct 17 10:00:37.946 2023] [541] rt: table rt: ramchunk saved ok (mode=periodic, last TID=1960091, current TID=1960883, ram=483.232 Mb, time delta=915 sec, took=15.344 sec)
[Tue Oct 17 10:02:02.535 2023] [561] rt: table rt_h: ramchunk saved ok (mode=periodic, last TID=1679093, current TID=1680134, ram=243.849 Mb, time delta=906 sec, took=6.828 sec)
[Tue Oct 17 10:02:08.297 2023] [567] rt: table rt_h: diskchunk 6968(17), segments 32 saved in 5.701673 (5.740652) sec, RAM saved/new 255003702/157454 ratio 0.950000 (soft limit 255013683, conf limit 268435456)
[Tue Oct 17 10:07:22.918 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,e298d353,47)) install timer expired
[Tue Oct 17 10:07:30.423 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,e298d353,47)) install timer expired
[Tue Oct 17 10:07:37.932 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,e298d353,48)) install timer expired
[Tue Oct 17 10:07:45.438 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,e298d353,48)) install timer expired
[Tue Oct 17 10:07:56.887 2023] [713] WARNING: Quorum: No node with complete state:
[Tue Oct 17 10:08:56.180 2023] [713] WARNING: Quorum: No node with complete state:
[Tue Oct 17 10:09:04.272 2023] [713] WARNING: Action message in non-primary configuration from member 3
[Tue Oct 17 10:09:32.610 2023] [694] WARNING: Failed to report last committed 1372995, -107 (Transport endpoint is not connected)
[Tue Oct 17 10:09:46.119 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,3f9be3c5,60)) install timer expired
[Tue Oct 17 10:09:49.455 2023] [713] WARNING: Action message in non-primary configuration from member 5
[Tue Oct 17 10:10:20.956 2023] [695] WARNING: last inactive check more than PT1.5S (3evs.inactive_check_period) ago (PT8.66193S), skipping check
[Tue Oct 17 10:10:29.481 2023] [695] WARNING: last inactive check more than PT1.5S (3evs.inactive_check_period) ago (PT4.82802S), skipping check
[Tue Oct 17 10:10:46.011 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,009ab359,69)) install timer expired
[Tue Oct 17 10:10:46.017 2023] [713] FATAL: /__w/manticoresearch/manticoresearch/build/galera-build/galera_populate-prefix/src/galera_populate/gcs/src/gcs_core.cpp:core_handle_uuid_msg():871: STATE EXCHANGE: failed for: 44452578-6cbc-11ee-97f5-0f6e9c0cf7f9: -107 (Transport endpoint is not connected)
[Tue Oct 17 10:10:54.516 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,e298d353,70)) install timer expired
[Tue Oct 17 10:10:59.521 2023] [695] WARNING: last inactive check more than PT1.5S (3evs.inactive_check_period) ago (PT1.62716S), skipping check
[Tue Oct 17 10:11:08.049 2023] [695] WARNING: last inactive check more than PT1.5S (3evs.inactive_check_period) ago (PT3.30301S), skipping check
[Tue Oct 17 10:11:11.060 2023] [713] WARNING: Quorum: No node with complete state:
[Tue Oct 17 10:12:03.720 2023] [695] WARNING: last inactive check more than PT1.5S (3evs.inactive_check_period) ago (PT3.08038S), skipping check
[Tue Oct 17 10:12:12.519 2023] [695] WARNING: last inactive check more than PT1.5S (3evs.inactive_check_period) ago (PT3.79157S), skipping check
[Tue Oct 17 10:12:12.707 2023] [713] WARNING: Quorum: No node with complete state:
[Tue Oct 17 10:12:40.042 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,009ab359,76)) install timer expired
[Tue Oct 17 10:12:48.598 2023] [695] WARNING: last inactive check more than PT1.5S (3evs.inactive_check_period) ago (PT6.04609S), skipping check
[Tue Oct 17 10:13:35.243 2023] [695] WARNING: last inactive check more than PT1.5S (3evs.inactive_check_period) ago (PT8.23923S), skipping check
[Tue Oct 17 10:13:49.965 2023] [695] WARNING: evs::proto(e298d353, GATHER, view_id(REG,3f9be3c5,85)) install timer expired
[Tue Oct 17 10:15:14.883 2023] [713] WARNING: Quorum: No node with complete state:
[Tue Oct 17 10:15:23.043 2023] [695] WARNING: last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.15792S), skipping check
[Tue Oct 17 10:16:17.134 2023] [527] rt: table rt: ramchunk saved ok (mode=periodic, last TID=1960883, current TID=1961310, ram=484.989 Mb, time delta=939 sec, took=39.188 sec)
[Tue Oct 17 10:17:17.348 2023] [526] rt: table rt_h: ramchunk saved ok (mode=periodic, last TID=1680141, current TID=1680503, ram=4.105 Mb, time delta=909 sec, took=14.813 sec)
[Tue Oct 17 10:17:17.594 2023] [526] rt: table rt_h: optimized progressive chunk(s) 33 ( left 16 ) in 15m 9.3s
[Tue Oct 17 10:18:06.799 2023] [695] WARNING: evs::proto(e298d353, INSTALL, view_id(REG,009ab359,100)) install timer expired
[Tue Oct 17 10:18:44.930 2023] [713] WARNING: Rejecting JOIN message from 4.0 (node_10.30.0.228_prodmain01_531): new State Transfer required.
[Tue Oct 17 10:19:18.193 2023] [714] FATAL: Local state seqno (1373631) is greater than group seqno (1373255): states diverged. Aborting to avoid potential data loss. Remove ‘/var/lib/manticore/grastate.dat’ file and restart if you wish to continue. (FATAL)
at /__w/manticoresearch/manticoresearch/build/galera-build/galera_populate-prefix/src/galera_populate/galera/src/replicator_str.cpp:state_transfer_required():39