Hi there ,
Below is the error when i am searching the data in manticore 6.0.2.
manticore keeps on crashing when some data is searched…
------ FATAL: CRASH DUMP -------
[Thu May 25 10:54:29.198 2023] [21895]
--- crashed SphinxQL request dump ---
SELECT *, RScheme as Scheme, RtShowPTR as RShowPtr, RtShowptrforallcompanies as RShowPtrForAllCompanies,
ProductNameRegEx as RegExProductName,AllowMOQ as AllowMOQ FROM idx_storeid_807,idx_storeid_817,
idx_storeid_829,idx_storeid_835,idx_storeid_838,idx_storeid_839,idx_storeid_841,idx_storeid_842,
idx_storeid_845,idx_storeid_846,idx_storeid_850,idx_storeid_859,idx_storeid_871,idx_storeid_882,
idx_storeid_884,idx_storeid_885,idx_storeid_886,idx_storeid_888,idx_storeid_892,idx_storeid_894,
idx_storeid_905,idx_storeid_911,idx_storeid_924,idx_storeid_934,idx_storeid_935,idx_storeid_940,
idx_storeid_961,idx_storeid_979,idx_storeid_997,idx_storeid_1018,idx_storeid_1038,
idx_storeid_1056,idx_storeid_1107,idx_storeid_1140,idx_storeid_1180,idx_storeid_1185,
idx_storeid_1226,idx_storeid_1644,idx_storeid_1800,idx_storeid_2027,idx_storeid_2222,
idx_storeid_2263,idx_storeid_2277,idx_storeid_2287,idx_storeid_2400,idx_storeid_2526,
idx_storeid_2825,idx_storeid_2860,idx_storeid_2999,idx_storeid_3078,idx_storeid_3090,
idx_storeid_3943,idx_storeid_3950,idx_storeid_4154,idx_storeid_4159,idx_storeid_4187,
idx_storeid_4202,idx_storeid_4217,idx_storeid_4225,idx_storeid_4331,idx_storeid_4363,
idx_storeid_4365,idx_storeid_4396,idx_storeid_4399,idx_storeid_4480,idx_storeid_4584,
idx_storeid_4645,idx_storeid_4660,idx_storeid_4693,idx_storeid_4732,idx_storeid_4738,
idx_storeid_4970,idx_storeid_5083,idx_storeid_5499,idx_storeid_5509,idx_storeid_5613,
idx_storeid_6044,idx_storeid_6196,idx_storeid_6306,idx_storeid_6339,idx_storeid_6396,
idx_storeid_6468,idx_storeid_6652,idx_storeid_6713,idx_storeid_6736,idx_storeid_6768,
idx_storeid_7302,idx_storeid_7306,idx_storeid_7542,idx_storeid_7546,idx_storeid_8060,
idx_storeid_8149,idx_storeid_8237,idx_storeid_8368,idx_storeid_8578,idx_storeid_8631,
idx_storeid_8694,idx_storeid_8734,idx_storeid_8752,idx_storeid_8875,idx_storeid_8883,
idx_storeid_8885,idx_storeid_8945,idx_storeid_9226,idx_storeid_9618,idx_storeid_10127,
idx_storeid_10259,idx_storeid_10400,idx_storeid_10487,idx_storeid_10637,idx_storeid_10838,
idx_storeid_10895,idx_storeid_11415 where prproductid > 0 and prproductId in(661783,
854836,1160098) limit 1000
--- request dump end ---
--- local index:idx_storeid_4331
Manticore 6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)
Handling signal 6
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.4
Configured with flags: Configured with these definitions: -DDISTR_BUILD=rhel7 -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (rhel7) (cross-compiled)
Stack bottom = 0x7f7b8c5c9220, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7f7b8c5d0000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x55da44fe208a]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x55da44ea4fd5]
/lib64/libpthread.so.0(+0xf630)[0x7f7bd9968630]
/lib64/libc.so.6(gsignal+0x37)[0x7f7bd95c1387]
/lib64/libc.so.6(abort+0x148)[0x7f7bd95c2a78]
/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x7f7bd9deba95]
/lib64/libstdc++.so.6(+0x5ea06)[0x7f7bd9de9a06]
/lib64/libstdc++.so.6(+0x5ea33)[0x7f7bd9de9a33]
/lib64/libstdc++.so.6(+0x5ec53)[0x7f7bd9de9c53]
/lib64/libstdc++.so.6(_ZSt20__throw_length_errorPKc+0x77)[0x7f7bd9e3ea67]
/usr/share/manticore/modules/lib_manticore_secondary.so(+0x100f62)[0x7f7bdb25ff62]
/usr/share/manticore/modules/lib_manticore_secondary.so(_ZN2SI13BlockReader_c20CreateBlocksIteratorERKNS_11BlockIter_tERSt6vectorIPN6common15BlockIterator_iESaIS7_EEPNS_16BitmapIterator_iE+0x5c)[0x7f7bdb1dfcbc]
/usr/share/manticore/modules/lib_manticore_secondary.so(_ZN2SI13BlockReader_c20CreateBlocksIteratorERKSt6vectorINS_11BlockIter_tESaIS2_EERS1_IPN6common15BlockIterator_iESaIS9_EE+0x81)[0x7f7bdb1dfff1]
/usr/share/manticore/modules/lib_manticore_secondary.so(_ZNK2SI16SecondaryIndex_c11GetValsRowsEPSt6vectorIPN6common15BlockIterator_iESaIS4_EERKNS2_8Filter_tEPKNS2_12RowidRange_tEjli+0x358)[0x7f7bdb1e6428]
/usr/share/manticore/modules/lib_manticore_secondary.so(_ZNK2SI16SecondaryIndex_c15CreateIteratorsERSt6vectorIPN6common15BlockIterator_iESaIS4_EERKNS2_8Filter_tEPKNS2_12RowidRange_tEjliRSs+0xb2)[0x7f7bdb1e6e42]
/usr/bin/searchd(_ZN19SIIteratorCreator_c17CreateSIIteratorsERSt6vectorIPN6common15BlockIterator_iESaIS3_EERK18CSphFilterSettingsl+0xdf)[0x55da452f585f]
/usr/bin/searchd(_ZN19SIIteratorCreator_c6CreateEv+0xaa)[0x55da452f5d3a]
/usr/bin/searchd(_Z28CreateSecondaryIndexIteratorPKN2SI7Index_iERN3sph8Vector_TI20SecondaryIndexInfo_tNS3_13DefaultCopy_TIS5_EENS3_14DefaultRelimitENS3_16DefaultStorage_TIS5_EEEERKNS4_I18CSphFilterSettingsNS6_ISD_EES8_NS9_ISD_EEEE13ESphCollationRK10ISphSchemaji+0xc2)[0x55da452f6372]
/usr/bin/searchd(_ZNK13CSphIndex_VLN14SpawnIteratorsERK9CSphQueryR16CSphQueryContextR21CreateFilterContext_tRK10ISphSchemaR19CSphQueryResultMetaiiRN3sph8Vector_TI18CSphFilterSettingsNSC_13DefaultCopy_TISE_EENSC_14DefaultRelimitENSC_16DefaultStorage_TISE_EEEE+0xfb)[0x55da44f435db]
/usr/bin/searchd(_ZNK13CSphIndex_VLN9MultiScanER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgsl+0x89a)[0x55da44f4477a]
/usr/bin/searchd(_ZNK13CSphIndex_VLN10MultiQueryER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgs+0x2bc)[0x55da44f4ee9c]
/usr/bin/searchd(_ZNK13CSphIndex_VLN12MultiQueryExEiPK9CSphQueryP15CSphQueryResultPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x59)[0x55da44f4f7d9]
/usr/bin/searchd(+0x862b2b)[0x55da44f06b2b]
/usr/bin/searchd(+0xd0b622)[0x55da453af622]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEE11VecTraits_TIhEENUlN5boost7context6detail10transfer_tEE_8__invokeES9_+0x1c)[0x55da457a8a2c]
/usr/bin/searchd(make_fcontext+0x2f)[0x55da457c776f]
Trying boost backtrace:
0# sphBacktrace(int, bool) in /usr/bin/searchd
1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
2# 0x00007F7BD9968630 in /lib64/libpthread.so.0
3# gsignal in /lib64/libc.so.6
4# abort in /lib64/libc.so.6
5# __gnu_cxx::__verbose_terminate_handler() in /lib64/libstdc++.so.6
6# 0x00007F7BD9DE9A06 in /lib64/libstdc++.so.6
7# 0x00007F7BD9DE9A33 in /lib64/libstdc++.so.6
8# 0x00007F7BD9DE9C53 in /lib64/libstdc++.so.6
9# std::__throw_length_error(char const*) in /lib64/libstdc++.so.6
10# 0x00007F7BDB25FF62 in /usr/share/manticore/modules/lib_manticore_secondary.so
11# SI::BlockReader_c::CreateBlocksIterator(SI::BlockIter_t const&, std::vector<common::BlockIterator_i, std::allocator<common::BlockIterator_i*> >&, SI::BitmapIterator_i) in /usr/share/manticore/modules/lib_manticore_secondary.so
12# SI::BlockReader_c::CreateBlocksIterator(std::vector<SI::BlockIter_t, std::allocator<SI::BlockIter_t> > const&, std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&) in /usr/share/manticore/modules/lib_manticore_secondary.so
13# SI::SecondaryIndex_c::GetValsRows(std::vector<common::BlockIterator_i, std::allocator<common::BlockIterator_i*> >, common::Filter_t const&, common::RowidRange_t const*, unsigned int, long, int) const in /usr/share/manticore/modules/lib_manticore_secondary.so
14# SI::SecondaryIndex_c::CreateIterators(std::vector<common::BlockIterator_i, std::allocator<common::BlockIterator_i*> >&, common::Filter_t const&, common::RowidRange_t const, unsigned int, long, int, std::string&) const in /usr/share/manticore/modules/lib_manticore_secondary.so
15# SIIteratorCreator_c::CreateSIIterators(std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&, CSphFilterSettings const&, long) in /usr/bin/searchd
16# SIIteratorCreator_c::Create() in /usr/bin/searchd
17# CreateSecondaryIndexIterator(SI::Index_i const*, sph::Vector_T<SecondaryIndexInfo_t, sph::DefaultCopy_T<SecondaryIndexInfo_t>, sph::DefaultRelimit, sph::DefaultStorage_T<SecondaryIndexInfo_t> >&, sph::Vector_T<CSphFilterSettings, sph::DefaultCopy_T<CSphFilterSettings>, sph::DefaultRelimit, sph::DefaultStorage_T<CSphFilterSettings> > const&, ESphCollation, ISphSchema const&, unsigned int, int) in /usr/bin/searchd
18# CSphIndex_VLN::SpawnIterators(CSphQuery const&, CSphQueryContext&, CreateFilterContext_t&, ISphSchema const&, CSphQueryResultMeta&, int, int, sph::Vector_T<CSphFilterSettings, sph::DefaultCopy_T<CSphFilterSettings>, sph::DefaultRelimit, sph::DefaultStorage_T<CSphFilterSettings> >&) const in /usr/bin/searchd
19# CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const in /usr/bin/searchd
20# CSphIndex_VLN::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in /usr/bin/searchd
21# CSphIndex_VLN::MultiQueryEx(int, CSphQuery const, CSphQueryResult, ISphMatchSorter**, CSphMultiQueryArgs const&) const in /usr/bin/searchd
22# 0x000055DA44F06B2B in /usr/bin/searchd
23# 0x000055DA453AF622 in /usr/bin/searchd
24# Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T<unsigned char>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
25# make_fcontext in /usr/bin/searchd
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
thd 0 (work_20), proto mysql, state query, command select
--- Totally 2 threads, and 1 client-working threads ---
------- CRASH DUMP END -------
[Thu May 25 10:54:35.803 2023] [22985] watchdog: main process 21895 crashed via CRASH_EXIT (exit code 2), will be restarted
[Thu May 25 10:54:35.809 2023] [22985] watchdog: main process 22636 forked ok
[Thu May 25 10:54:36.180 2023] [22636] starting daemon version '6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)' ...
[Thu May 25 10:54:36.180 2023] [22636] listening on all interfaces for mysql, port=9306
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.001
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.001; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.002
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.002; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.003
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.003; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.004
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.004; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.005
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.005; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.006
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.006; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.007
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.007; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.008
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.008; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.009
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.009; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.010
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.010; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.011
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.011; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.012
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.012; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.013
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.013; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.014
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.014; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.015
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.015; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.016
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.016; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: replaying log /var/lib/manticore/data//binlog.017
[Thu May 25 10:55:17.480 2023] [22638] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying /var/lib/manticore/data//binlog.017; 0.0 MB in 0.000 sec
[Thu May 25 10:55:17.480 2023] [22638] binlog: finished replaying total 17 in 0.000 sec
[Thu May 25 10:55:17.482 2023] [22639] prereading 11491 tables
[Thu May 25 10:55:17.482 2023] [22636] WARNING: [BUDDY] no SPHINX or HTTP listeners found, disabled
[Thu May 25 10:55:17.482 2023] [22636] accepting connections
[Thu May 25 10:55:34.047 2023] [22639] preread 11491 tables in 16.565 sec