Имеется кластер manticore из 3 нод. При выполнении запроса на одной из нод
ALTER CLUSTER esz DROP pupils;
ALTER TABLE pupils DROP COLUMN statusesinorg;
ALTER TABLE pupils ADD COLUMN statusesinorg multi64;
ALTER CLUSTER esz ADD pupils;
индекс pupils стал падать с ошибкой:
------- FATAL: CRASH DUMP -------
[Fri Apr 21 18:24:07.979 2023] [ 1676]
--- crashed SphinxQL request dump ---
REPLACE INTO esz:pupils (Id,FullName,FirstName,LastName,MiddleName,ContingentLinkTypeId,
ContingentGuid,BirthDate,HasGroups,HasInContingent,EducationTypeId,SexName,SexId,ServiceCount,
RMSPPrivilegeData,PrivilegeCategories,MegaRelationStatuses,ServiceIds,OrganizationIds,
VedomstvoIds,StatusesInOrg,StatusesInVedom,RbndoClassificationIds) VALUES(1315955,
'Ð<9c>------а Ð<92>аленÑ<82>ина Ð<95>--------а','Ð<92>аленÑ<82>ина','Ð<9c>------а','Ð<95>--------а',
1,'b8f6ad02-f4f3-4be1-9889-10ad323ddf15',633253248000000000,1,1,1,'Ð<96>енÑ<81>кий',
2,1,'{\"DateEnrollment\":[17045,17420,17805,18871],\"PrivilegeCount\":0}','','[{\"vedId\":1,
\"orgId\":3790,\"status\":2,\"cnt\":3},{\"vedId\":1,\"orgId\":3790,\"status\":1,\"cnt\":1}]',
(133972,384900,591230,1424464),(3790,3790,3790,3790),(1,1,1,1),(379000002,379000002,
379000002,379000001),(12,12,12,11),(1010127,1010127,1010127,1010441,1010441,1010441,
1010141,1010141,1010141,1010461,1010461,1010461))
--- request dump end ---
--- local index:0²N^]s^?
Manticore 6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.4
Configured with flags: Configured with these definitions: -DDISTR_BUILD=rhel7 -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (rhel7) (cross-compiled)
Stack bottom = 0x7f6efa2a2050, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7f6efa2a0000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x55a1657ff08a]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x55a1656c1fd5]
/lib64/libpthread.so.0(+0xf630)[0x7f732a2b4630]
/usr/bin/searchd(_ZN13CSphIndex_VLN10MergeWordsI16DiskIndexQword_cILb1ELb0EES2_EEbPKS_S4_11VecTraits_TIjES6_P14CSphHitBuilderR10CSphStringR17CSphIndexProgress+0xc2f)[0x55a1657b875f]
/usr/bin/searchd(_ZN13CSphIndex_VLN7DoMergeEPKS_S1_PK10ISphFilterR10CSphStringR17CSphIndexProgressbb+0x626)[0x55a16575aad6]
/usr/bin/searchd(_Z8sphMergePK9CSphIndexS1_11VecTraits_TI18CSphFilterSettingsER17CSphIndexProgressR10CSphString+0x72)[0x55a16575b482]
/usr/bin/searchd(_ZN9RtIndex_c15MergeDiskChunksEPKcRK17CSphRefcountedPtrIK11DiskChunk_cES7_R17CSphIndexProgress11VecTraits_TI18CSphFilterSettingsE+0x65)[0x55a165abf0e5]
/usr/bin/searchd(_ZN9RtIndex_c14MergeTwoChunksEiiPi+0x496)[0x55a165ac2ef6]
/usr/bin/searchd(_ZN9RtIndex_c19ProgressiveOptimizeEi+0x597)[0x55a165ac4387]
/usr/bin/searchd(_ZN9RtIndex_c8OptimizeE14OptimizeTask_t+0xed)[0x55a165ac3a9d]
/usr/bin/searchd(+0x755d07)[0x55a165616d07]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEE11VecTraits_TIhEENUlN5boost7context6detail10transfer_tEE_8__invokeES9_+0x1c)[0x55a165fc5a2c]
/usr/bin/searchd(make_fcontext+0x2f)[0x55a165fe476f]
Trying boost backtrace:
0# sphBacktrace(int, bool) in /usr/bin/searchd
1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
2# 0x00007F732A2B4630 in /lib64/libpthread.so.0
3# bool CSphIndex_VLN::MergeWords<DiskIndexQword_c<true, false>, DiskIndexQword_c<true, false> >(CSphIndex_VLN const*, CSphIndex_VLN const*, VecTraits_T<unsigned int>, VecTraits_T<unsigned int>, CSphHitBuilder*, CSphString&, CSphIndexProgress&) in /usr/bin/searchd
4# CSphIndex_VLN::DoMerge(CSphIndex_VLN const*, CSphIndex_VLN const*, ISphFilter const*, CSphString&, CSphIndexProgress&, bool, bool) in /usr/bin/searchd
5# sphMerge(CSphIndex const*, CSphIndex const*, VecTraits_T<CSphFilterSettings>, CSphIndexProgress&, CSphString&) in /usr/bin/searchd
6# RtIndex_c::MergeDiskChunks(char const*, CSphRefcountedPtr<DiskChunk_c const> const&, CSphRefcountedPtr<DiskChunk_c const> const&, CSphIndexProgress&, VecTraits_T<CSphFilterSettings>) in /usr/bin/searchd
7# RtIndex_c::MergeTwoChunks(int, int, int*) in /usr/bin/searchd
8# RtIndex_c::ProgressiveOptimize(int) in /usr/bin/searchd
9# RtIndex_c::Optimize(OptimizeTask_t) in /usr/bin/searchd
10# 0x000055A165616D07 in /usr/bin/searchd
11# Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T<unsigned char>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
12# make_fcontext in /usr/bin/searchd
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
thd 0 (work_1), proto mysql, state query, command replace
--- Totally 3 threads, and 1 client-working threads ---
------- CRASH DUMP END -------
При проверке индекса оказалось, что он битый.
indextool --check pupils;
Manticore 6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
using config file '/etc/manticoresearch/manticore.conf'...
WARNING: key 'max_children' was permanently removed from configuration. Refer to documentation for details.
WARNING: key 'read_timeout' is deprecated in /etc/manticoresearch/manticore.conf line 27; use 'network_timeout' instead.
WARNING: key 'preopen_indexes' is deprecated in /etc/manticoresearch/manticore.conf line 28; use 'preopen_tables' instead.
WARNING: key 'workers' is deprecated in /etc/manticoresearch/manticore.conf line 31; use 'default value' instead.
checking table 'pupils'...
checking schema...
checking RT segment 0(26)...
checking rows...
checking dead row map...
checking RT segment 1(26)...
checking rows...
checking dead row map...
checking RT segment 2(26)...
checking rows...
checking dead row map...
checking RT segment 3(26)...
checking rows...
checking dead row map...
checking RT segment 4(26)...
checking rows...
checking dead row map...
checking RT segment 5(26)...
checking rows...
checking dead row map...
checking RT segment 6(26)...
checking rows...
checking dead row map...
checking RT segment 7(26)...
checking rows...
checking dead row map...
checking RT segment 8(26)...
checking rows...
checking dead row map...
checking RT segment 9(26)...
checking rows...
checking dead row map...
checking RT segment 10(26)...
checking rows...
checking dead row map...
checking RT segment 11(26)...
checking rows...
checking dead row map...
checking RT segment 12(26)...
checking rows...
checking dead row map...
checking RT segment 13(26)...
checking rows...
checking dead row map...
checking RT segment 14(26)...
checking rows...
checking dead row map...
checking RT segment 15(26)...
checking rows...
checking dead row map...
checking RT segment 16(26)...
checking rows...
checking dead row map...
checking RT segment 17(26)...
checking rows...
checking dead row map...
checking RT segment 18(26)...
checking rows...
checking dead row map...
checking RT segment 19(26)...
checking rows...
checking dead row map...
checking RT segment 20(26)...
checking rows...
checking dead row map...
checking RT segment 21(26)...
checking rows...
checking dead row map...
checking RT segment 22(26)...
checking rows...
checking dead row map...
checking RT segment 23(26)...
checking rows...
checking dead row map...
checking RT segment 24(26)...
checking rows...
checking dead row map...
checking RT segment 25(26)...
checking rows...
checking dead row map...
checking disk chunk, extension 426, 0(6)...
WARNING: failed to load stopwords from either 'stopwords.txt' or '/usr/share/manticore/stopwords/stopwords.txt'
FAILED, unable to open stopwords 'stopwords.txt': No such file or directory
checking schema...
checking dictionary...
FAILED, checkpoint count mismatch (read=697, calc=0)
checking data...
FAILED, rowid out of bounds (wordid=0(-), rowid=8592)
FAILED, rowid out of bounds (wordid=0(-), rowid=64363)
FAILED, rowid out of bounds (wordid=0(-), rowid=115855)
FAILED, rowid out of bounds (wordid=0(-), rowid=186369)
FAILED, rowid out of bounds (wordid=0(-), rowid=188320)
FAILED, rowid out of bounds (wordid=0(-), rowid=251586)
FAILED, rowid out of bounds (wordid=0(-), rowid=268558)
FAILED, rowid out of bounds (wordid=0(-), rowid=276445)
FAILED, rowid out of bounds (wordid=0(-), rowid=288415)
FAILED, rowid out of bounds (wordid=0(-), rowid=371928)
FAILED, rowid out of bounds (wordid=0(-), rowid=371938)
FAILED, rowid out of bounds (wordid=0(-), rowid=374612)
FAILED, rowid out of bounds (wordid=0(-), rowid=379493)
FAILED, rowid out of bounds (wordid=0(-), rowid=379739)
FAILED, rowid out of bounds (wordid=0(-), rowid=379742)
FAILED, rowid out of bounds (wordid=0(-), rowid=414023)
FAILED, rowid out of bounds (wordid=0(-), rowid=452080)
FAILED, rowid out of bounds (wordid=0(-), rowid=471316)
FAILED, rowid out of bounds (wordid=0(-), rowid=471327)
FAILED, rowid out of bounds (wordid=0(-), rowid=479998)
FAILED, rowid out of bounds (wordid=0(-), rowid=507641)
FAILED, rowid out of bounds (wordid=0(-), rowid=507652)
FAILED, rowid out of bounds (wordid=0(-), rowid=507665)
FAILED, rowid out of bounds (wordid=0(-), rowid=510987)
FAILED, rowid out of bounds (wordid=0(-), rowid=522796)
FAILED, rowid out of bounds (wordid=0(-), rowid=558994)
FAILED, rowid out of bounds (wordid=0(-), rowid=559181)
FAILED, rowid out of bounds (wordid=0(-), rowid=592203)
FAILED, rowid out of bounds (wordid=0(-), rowid=592403)
FAILED, rowid out of bounds (wordid=0(-), rowid=602244)
FAILED, rowid out of bounds (wordid=0(--), rowid=180788)
FAILED, rowid out of bounds (wordid=0(----), rowid=219541)
FAILED, rowid out of bounds (wordid=0(----), rowid=286579)
FAILED, rowid out of bounds (wordid=0(-----), rowid=174614)
FAILED, rowid out of bounds (wordid=0(-----), rowid=183392)
FAILED, rowid out of bounds (wordid=0(---------), rowid=187424)
FAILED, rowid out of bounds (wordid=0(----------), rowid=183050)
FAILED, rowid out of bounds (wordid=0(----------), rowid=184614)
FAILED, rowid out of bounds (wordid=0(----------), rowid=184621)
FAILED, rowid out of bounds (wordid=0(----------), rowid=184622)
FAILED, rowid out of bounds (wordid=0(----------ч), rowid=316587)
FAILED, rowid out of bounds (wordid=0(-али), rowid=217319)
FAILED, rowid out of bounds (wordid=0(-али), rowid=254874)
FAILED, rowid out of bounds (wordid=0(-эмилия), rowid=205068)
FAILED, rowid out of bounds (wordid=0(0028827c5dd569cad784b0b93200c864), rowid=173361)
FAILED, rowid out of bounds (wordid=0(144e4d79f05776a57d72408b9a149360), rowid=444067)
FAILED, rowid out of bounds (wordid=0(27f1de39de1ebb2a2731fb0921237675), rowid=173361)
FAILED, rowid out of bounds (wordid=0(28137a745ea12eebb2d5d5e181f33773), rowid=444067)
FAILED, rowid out of bounds (wordid=0(47205276673b8eaf1192ae22338d5022), rowid=173361)
FAILED, rowid out of bounds (wordid=0(9236bb9a7fdbc33468d67886b0206376), rowid=253240)
FAILED, rowid out of bounds (wordid=0(_----------_), rowid=187804)
FAILED, rowid out of bounds (wordid=0(_---------_), rowid=187754)
FAILED, rowid out of bounds (wordid=0(_--------_), rowid=187773)
FAILED, rowid out of bounds (wordid=0(_-----_), rowid=187823)
FAILED, rowid out of bounds (wordid=0(a0b201d732e580f6b014f5a7270197d9), rowid=253240)
FAILED, rowid out of bounds (wordid=0(b242ff362d58c6f4f29fc3325eeb9e86), rowid=253240)
FAILED, rowid out of bounds (wordid=0(bbc64d12c40ecc253fe579ab419fb50f), rowid=444067)
FAILED, rowid out of bounds (wordid=0(c-------а), rowid=190897)
FAILED, rowid out of bounds (wordid=0(c-------а), rowid=191053)
FAILED, rowid out of bounds (wordid=0(c-------ч), rowid=192923)
FAILED, rowid out of bounds (wordid=0(x--------ч), rowid=178943)
FAILED, rowid out of bounds (wordid=0(Е), rowid=180783)
FAILED, rowid out of bounds (wordid=0(Е), rowid=231213)
FAILED, rowid out of bounds (wordid=0(Е----------ч), rowid=52995)
FAILED, rowid out of bounds (wordid=0(Е----------ч), rowid=214100)
FAILED, rowid out of bounds (wordid=0(Е----------ч), rowid=219414)
FAILED, rowid out of bounds (wordid=0(Е---------а), rowid=3396)
FAILED, rowid out of bounds (wordid=0(Е---------а), rowid=29001)
FAILED, rowid out of bounds (wordid=0(Е---------а), rowid=40641)
FAILED, rowid out of bounds (wordid=0(Е---------а), rowid=65544)
FAILED, rowid out of bounds (wordid=0(Е---------а), rowid=69007)
FAILED, rowid out of bounds (wordid=0(Е---------а), rowid=310972)
FAILED, rowid out of bounds (wordid=0(Е---------ч), rowid=270129)
FAILED, rowid out of bounds (wordid=0(Е--------а), rowid=11683)
FAILED, rowid out of bounds (wordid=0(Е--------а), rowid=40641)
FAILED, rowid out of bounds (wordid=0(Е--------а), rowid=60814)
FAILED, rowid out of bounds (wordid=0(Е--------и), rowid=67345)
FAILED, rowid out of bounds (wordid=0(Е--------и), rowid=84273)
FAILED, rowid out of bounds (wordid=0(Е--------ч), rowid=71028)
FAILED, rowid out of bounds (wordid=0(Е--------ы), rowid=3270)
FAILED, rowid out of bounds (wordid=0(Е-------а), rowid=72553)
FAILED, rowid out of bounds (wordid=0(Е-------а), rowid=116184)
FAILED, rowid out of bounds (wordid=0(Е-------в), rowid=68700)
FAILED, rowid out of bounds (wordid=0(Е-------и), rowid=59714)
FAILED, rowid out of bounds (wordid=0(Е-------ч), rowid=64202)
FAILED, rowid out of bounds (wordid=0(Е-------ч), rowid=130043)
FAILED, rowid out of bounds (wordid=0(Е-------ч), rowid=217913)
FAILED, rowid out of bounds (wordid=0(Е-------ч), rowid=262582)
FAILED, rowid out of bounds (wordid=0(Е-------ч), rowid=476826)
FAILED, rowid out of bounds (wordid=0(Е------а), rowid=219684)
FAILED, rowid out of bounds (wordid=0(Е------а), rowid=517025)
FAILED, rowid out of bounds (wordid=0(Е------ч), rowid=614249)
FAILED, rowid out of bounds (wordid=0(Е-----а), rowid=20104)
FAILED, rowid out of bounds (wordid=0(Е-----а), rowid=243737)
FAILED, rowid out of bounds (wordid=0(Е-----а), rowid=412190)
FAILED, rowid out of bounds (wordid=0(Е-----в), rowid=172410)
FAILED, rowid out of bounds (wordid=0(Е-----в), rowid=231861)
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check FAILED, 99 of 4512316 failures reported, 0.5 sec elapsed
checking disk chunk, extension 427, 1(6)...
WARNING: failed to load stopwords from either 'stopwords.txt' or '/usr/share/manticore/stopwords/stopwords.txt'
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check FAILED, 99 of 8422478 failures reported, 1.0 sec elapsed
checking disk chunk, extension 428, 2(6)...
WARNING: failed to load stopwords from either 'stopwords.txt' or '/usr/share/manticore/stopwords/stopwords.txt'
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check FAILED, 99 of 12948515 failures reported, 1.6 sec elapsed
checking disk chunk, extension 425, 3(6)...
WARNING: failed to load stopwords from either 'stopwords.txt' or '/usr/share/manticore/stopwords/stopwords.txt'
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check FAILED, 99 of 18151104 failures reported, 2.2 sec elapsed
checking disk chunk, extension 429, 4(6)...
WARNING: failed to load stopwords from either 'stopwords.txt' or '/usr/share/manticore/stopwords/stopwords.txt'
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check FAILED, 99 of 18151105 failures reported, 7.4 sec elapsed
checking disk chunk, extension 430, 5(6)...
WARNING: failed to load stopwords from either 'stopwords.txt' or '/usr/share/manticore/stopwords/stopwords.txt'
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check FAILED, 99 of 18151106 failures reported, 14.1 sec elapsed
check FAILED, 99 of 18151106 failures reported, 14.1 sec elapsed
Скрипт индекса pupils:
CREATE TABLE pupils (
id bigint,
sexid integer,
hasgroups integer,
hasincontingent integer,
contingentlinktypeid integer,
educationtypeid integer,
servicecount integer,
birthdate bigint,
fullname string attribute indexed,
firstname string attribute,
lastname string attribute,
middlename string attribute,
sexname string attribute,
contingentguid string attribute,
vedomstvoids multi,
organizationids multi,
serviceids multi,
rbndoclassificationids multi,
rmspprivilegedata json,
privilegecategories json,
megarelationstatuses json,
statusesinvedom multi,
statusesinorg multi64
) min_infix_len='2' index_exact_words='1' charset_table='0..9, A..Z->a..z, -, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451->U+0435, U+401->U+0415' blend_chars='U+0060,U+0027,U+2019,U+2018' morphology='stem_enru,soundex' stopwords='/var/lib/manticore/pupils/stopwords.txt' rt_mem_limit='536870912'
OS: Red Hat Enterprise Linux Server release 7.9 (Maipo)
Manticore: 6.0.2_230210.89c7a5139-1