I am trying to add a local rt index to a cluster. The index has about 15M records. When I run alter cluster, add a local rt index using a local mysql client I get query timed out after 5 minutes. Is there a way to increase this timeout or another way I should be adding a large index to a cluster?
MySQL [(none)]> alter cluster MY_cluster add myindex;
ERROR 1064 (42000): ‘10.0.4.5:9312’: query timed out
you could use searchd options such as network_timeout and sphinxql_timeout or mysql client specific options as it is not clean who closes connection - daemon or client.
Thanks. I will try those settings. It seems that sphinx_timeout defaults to 15 minutes and mysql client might be set to 8 hours by default.
I also tried re-creating the cluster, adding the index into the cluster and joining another cluster node. That also timed out and eventually crashed manticore searchd.
@Damir_Tresnjo We’ll appreciate if you specify how to reproduce it (or at least provide more info about the case: logs etc.) and create an issue on github - Sign in to GitHub · GitHub
I am not sure exactly how to reproduce the state when searchd starts crashing. It is definitely related to a cluster setup. We have a two node cluster, restart one of the nodes a few times and it gets into this state where every second query to an index will crash it consistently. We will try to figure out the exact steps and see if we can reproduce it. Here is one of the crash logs, not sure if that helps. We have seen this issue with the latest dev build as well.
------- FATAL: CRASH DUMP -------
[Thu Aug 27 22:56:12.882 2020] [24939]
— crashed HTTP request dump —
select * from dx1privateparts limit 2
— request dump end —
— local index:dx1partsinventory
Manticore 3.5.0 1d34c49@200722 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=rhel7 -DUSE_SSL=ON -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_E
XPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share
/manticore -DUSE_RE2=1 -DUSE_ICU=1 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1
-DWITH_ZLIB=ON -DGALERA_SONAME=libgalera_manticore.so.31 -DSYSCONFDIR=/etc/manticoresearch
Host OS is Linux runner-fa6cab46-project-3858465-concurrent-0 4.19.78-coreos #1 SMP Mon Oct 14 22:56:39 -00 2019 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f5024061420, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x591bd0)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x591bd0, stack=0x7f5024060000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x90)[0x723f30]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x1b2)[0x592182]
/lib64/libpthread.so.0(+0xf630)[0x7f5184762630]
/usr/bin/searchd[0x8cff00]
/usr/bin/searchd(_Z14sphGetBlobAttrRK9CSphMatchRK15CSphAttrLocatorPKh+0x22)[0x8d15d2]
/usr/bin/searchd(_ZN20MatchesToNewSchema_c7ProcessEP9CSphMatch+0x1f2)[0x749852]
/usr/bin/searchd(_ZN14CSphMatchQueueI16MatchGeneric2_fnLb0EE8FinalizeER18ISphMatchProcessorb+0x5d)[0x7c9f5d]
/usr/bin/searchd(_ZN15ISphMatchSorter30TransformPooled2StandalonePtrsESt8functionIFPKhPK9CSphMatchEE+0x2c6)[0x732216]
/usr/bin/searchd[0x87397c]
/usr/bin/searchd(_ZNK9RtIndex_c10MultiQueryEPK9CSphQueryP15CSphQueryResultiPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x10f8)[0x891ef8]
/usr/bin/searchd(_ZNK9RtIndex_c12MultiQueryExEiPK9CSphQueryPP15CSphQueryResultPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x77)[0x893427]
/usr/bin/searchd(_ZN15SearchHandler_c16RunLocalSearchesEv+0x62b)[0x5d0f8b]
/usr/bin/searchd(_ZN15SearchHandler_c9RunSubsetEii+0xdbb)[0x5e089b]
/usr/bin/searchd(_ZN15SearchHandler_c10RunQueriesEv+0xae)[0x5e157e]
/usr/bin/searchd(_ZN19HttpSearchHandler_c7ProcessEv+0xda)[0x614e4a]
/usr/bin/searchd[0x60f92a]
/usr/bin/searchd(_Z17sphLoopClientHttpPKhiRN3sph8Vector_TIhNS1_13DefaultCopy_TIhEENS1_14DefaultRelimitENS1_16DefaultStorage_TIhEEEE+0x1c4)[0x60fe94]
/usr/bin/searchd(_Z9HttpServe11SharedPtr_TIP16AsyncNetBuffer_c9Deleter_TIS1_L5ETYPE0EE16ISphRefcountedMTE+0x28f)[0x667c0f]
/usr/bin/searchd(_Z10MultiServe11SharedPtr_TIP13SockWrapper_c9Deleter_TIS1_L5ETYPE0EE16ISphRefcountedMTESt4pairIitE+0xb5)[0x665ea5]
/usr/bin/searchd[0x666c6e]
/usr/bin/searchd(ZZN7Threads11CoRoutine_cC1ESt8functionIFvvEEmENUlN5boost7context6detail10transfer_tEE_4_FUNES7+0x17)[0x95cc07]
/usr/bin/searchd(make_fcontext+0x2f)[0x96218f]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (Issues · manticoresoftware/manticoresearch · GitHub)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter ‘Reporting bugs’ in the documentation
(Manticore Search Manual: Reporting bugs)
Dump with GDB via watchdog
[Thu Aug 27 22:56:13.049 2020] [27472] watchdog: got USR1, performing dump of child’s stack
Will run gdb on ‘/usr/bin/searchd’, pid ‘24939’
Error reading attached process’s symbol file.
: No such file or directory.
Error reading attached process’s symbol file.
: No such file or directory.
[New LWP 24974]
[New LWP 24944]
[New LWP 24943]
[New LWP 24942]
[New LWP 24941]
[New LWP 24940]
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib64/libthread_db.so.1”.
0x00007f518363d9a3 in select () from /lib64/libc.so.6
Id Target Id Frame
7 Thread 0x7f518524f700 (LWP 24940) “work_0” 0x00007f518363d9a3 in select () from /lib64/libc.so.6
6 Thread 0x7f518522e700 (LWP 24941) “work_1” 0x00007f518475ea35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
5 Thread 0x7f518520d700 (LWP 24942) “TaskSched” 0x00007f518475ede2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
4 Thread 0x7f50a4a6d700 (LWP 24943) “TaskW_1” 0x00007f518475ede2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 0x7f502a69a700 (LWP 24944) “TickPool_0” 0x00007f5183646eb3 in epoll_wait () from /lib64/libc.so.6
2 Thread 0x7f5028712700 (LWP 24974) “sphTimer” 0x00007f518475ea35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
Thread 7 (Thread 0x7f518524f700 (LWP 24940)):
#0 0x00007f518363d9a3 in select () from /lib64/libc.so.6
#1 0x00000000006704f4 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x00000000000bdb74 in ?? ()
#4 0x00000000000000f5 in ?? ()
#5 0x0000000000723fb0 in ?? ()
#6 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7f518522e700 (LWP 24941)):
#0 0x00007f518475ea35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000093f9d4 in ?? ()
#2 0x00007f518522dc5f in ?? ()
#3 0x00000000015198a0 in ?? ()
#4 0x0000000001519908 in ?? ()
#5 0x0000000000000001 in ?? ()
#6 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7f518520d700 (LWP 24942)):
#0 0x00007f518475ede2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000072a9e6 in ?? ()
#2 0x000000005f4840ce in ?? ()
#3 0x0000000033540ca4 in ?? ()
#4 0x00000000001b6257 in ?? ()
#5 0x0000000000e6de00 in ?? ()
#6 0x00000000001b6257 in ?? ()
#7 0x000000006af824fa in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7f50a4a6d700 (LWP 24943)):
#0 0x00007f518475ede2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000072a896 in ?? ()
#2 0x000000005f483c24 in ?? ()
#3 0x000000000d6e3e28 in ?? ()
#4 0x00000000015072d0 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f502a69a700 (LWP 24944)):
#0 0x00007f5183646eb3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000060556a in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000001514850 in ?? ()
#4 0x00000000ffffffff in ?? ()
#5 0x0000000000000001 in ?? ()
#6 0x00007f502a699b48 in ?? ()
#7 0x0000000000664b41 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f5028712700 (LWP 24974)):
#0 0x00007f518475ea35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000072aa5b in ?? ()
#2 0x0000000000e796c0 in ?? ()
#3 0x00007f518363d9af in select () from /lib64/libc.so.6
#4 0x0000000000e796a8 in ?? ()
#5 0x0000000000e796a0 in ?? ()
#6 0x0000000000e796c0 in ?? ()
#7 0x0000000000021000 in ?? ()
#8 0x0000000000960cfa in ?? ()
#9 0x00007f5053a44d60 in ?? ()
#10 0x00007f5028711d3f in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f5185251900 (LWP 24939)):
#0 0x00007f518363d9a3 in select () from /lib64/libc.so.6
#1 0x00000000006704f4 in ?? ()
#2 0x0000000000000000 in ?? ()
Main thread:
#0 0x00007f518363d9a3 in select () from /lib64/libc.so.6
#1 0x00000000006704f4 in ?? ()
#2 0x0000000000000000 in ?? ()
Local variables:
No symbol table info available.
[Inferior 1 (process 24939) detached]
— active threads —
thd 0 (work_0), proto http, state net_read, command -
— Totally 2 threads, and 1 client-working threads —
------- CRASH DUMP END -------
seems your index got broken, could you create index at Github and upload index data to our FTP?
I created a new VM to reproduce the issue. I imported data and wanted to remove some customer sensitive data from the index before I sent it to you but running an update command on the index crashed searchd. So, I cannot send you data before I can replace it with some dummy values and I cannot update the index because it is crashing. I will need to see if we can change our ETL process to load dummy data. I run indextool after the crash and the results are clean (output at the bottom)
------- FATAL: CRASH DUMP -------
[Fri Aug 28 17:11:13.923 2020] [ 6255]
— crashed SphinxQL request dump —
update dx1parts set dealershipid=‘0’ where dealershipid <> ‘0’
— request dump end —
— local index:dx1parts
Manticore 3.5.0 1d34c49@200722 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=rhel7 -DUSE_SSL=ON -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_RE2=1 -DUSE_ICU=1 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SONAME=libgalera_manticore.so.31 -DSYSCONFDIR=/etc/manticoresearch
Host OS is Linux runner-fa6cab46-project-3858465-concurrent-0 4.19.78-coreos #1 SMP Mon Oct 14 22:56:39 -00 2019 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f52b4020b90, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x591b10)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x591b10, stack=0x7f52b4020000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x90)[0x723f30]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x1b2)[0x592182]
/lib64/libpthread.so.0(+0xf630)[0x7f53cd023630]
/usr/bin/searchd[0x8cfe20]
/usr/bin/searchd(_ZNK9CSphMatch13FetchAttrDataERK15CSphAttrLocatorPKhRi+0x14)[0x677d24]
/usr/bin/searchd(_ZNK14FilterString_c4EvalERK9CSphMatch+0x25)[0x847d75]
/usr/bin/searchd(_ZNK10Filter_Not4EvalERK9CSphMatch+0xe)[0x84591e]
/usr/bin/searchd(_ZNK13CSphIndex_VLN12ScanByBlocksERK16CSphQueryContextP15CSphQueryResultiPP15ISphMatchSorterR9CSphMatchibbil+0x478)[0x690278]
/usr/bin/searchd(_ZNK13CSphIndex_VLN9MultiScanEPK9CSphQueryP15CSphQueryResultiPP15ISphMatchSorterRK18CSphMultiQueryArgs+0xaf0)[0x6aac00]
/usr/bin/searchd(_ZNK13CSphIndex_VLN10MultiQueryEPK9CSphQueryP15CSphQueryResultiPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x6b8)[0x6ba9a8]
/usr/bin/searchd[0x883fa3]
/usr/bin/searchd[0x9607e7]
/usr/bin/searchd(_ZN7Threads10CoExecuteNEOSt8functionIFvvEEi+0x17f)[0x95c57f]
/usr/bin/searchd(_ZNK9RtIndex_c10MultiQueryEPK9CSphQueryP15CSphQueryResultiPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x9e3)[0x8917e3]
/usr/bin/searchd(_ZNK9RtIndex_c12MultiQueryExEiPK9CSphQueryPP15CSphQueryResultPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x77)[0x893427]
/usr/bin/searchd(_ZN15SearchHandler_c16RunLocalSearchesEv+0x62b)[0x5d0f8b]
/usr/bin/searchd(ZN15SearchHandler_c14RunActionQueryERK9CSphQueryRK10CSphStringPS3+0x1c8)[0x5dc6f8]
/usr/bin/searchd(_Z25HandleMySqlExtendedUpdateR14AttrUpdateArgs+0x7c)[0x5dcbcc]
/usr/bin/searchd(_ZN15CommitMonitor_c6UpdateER10CSphString+0xb8)[0x617338]
/usr/bin/searchd[0x626452]
/usr/bin/searchd[0x5b771d]
/usr/bin/searchd(Z20sphHandleMysqlUpdateR19StmtErrorReporter_iRK9SqlStmt_tRK10CSphStringRS4+0x213)[0x5de703]
/usr/bin/searchd(_ZN16CSphinxqlSession7ExecuteERK10CSphStringR11RowBuffer_i+0x7d6)[0x5fe916]
/usr/bin/searchd[0x66a6d8]
/usr/bin/searchd(_Z8SqlServe11SharedPtr_TIP13SockWrapper_c9Deleter_TIS1_L5ETYPE0EE16ISphRefcountedMTE+0x8e8)[0x66b488]
/usr/bin/searchd[0x666b2a]
/usr/bin/searchd(ZZN7Threads11CoRoutine_cC1ESt8functionIFvvEEmENUlN5boost7context6detail10transfer_tEE_4_FUNES7+0x17)[0x95cc07]
/usr/bin/searchd(make_fcontext+0x2f)[0x96218f]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (Issues · manticoresoftware/manticoresearch · GitHub)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter ‘Reporting bugs’ in the documentation
Dump with GDB via watchdog
[Fri Aug 28 17:11:13.998 2020] [6254] watchdog: got USR1, performing dump of child’s stack
Will run gdb on ‘/usr/bin/searchd’, pid ‘6255’
Error reading attached process’s symbol file.
: No such file or directory.
Error reading attached process’s symbol file.
: No such file or directory.
[New LWP 6565]
[New LWP 6430]
[New LWP 6429]
[New LWP 6428]
[New LWP 6427]
[New LWP 6396]
[New LWP 6383]
[New LWP 6258]
[New LWP 6257]
[New LWP 6256]
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib64/libthread_db.so.1”.
0x00007f53cbefe9a3 in select () from /lib64/libc.so.6
Id Target Id Frame
11 Thread 0x7f53cdb10700 (LWP 6256) “work_0” 0x00007f53cd01fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
10 Thread 0x7f53cdaef700 (LWP 6257) “work_1” 0x00007f53cbefe9a3 in select () from /lib64/libc.so.6
9 Thread 0x7f53cdace700 (LWP 6258) “TaskSched” 0x00007f53cd01fde2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
8 Thread 0x7f52edd6f700 (LWP 6383) “TaskW_1” 0x00007f53cd01fde2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
7 Thread 0x7f52b9f47700 (LWP 6396) “TickPool_0” 0x00007f53cbf07eb3 in epoll_wait () from /lib64/libc.so.6
6 Thread 0x7f52b3fff700 (LWP 6427) “work_0” 0x00007f53cd01fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
5 Thread 0x7f52b1ffb700 (LWP 6428) “work_0” 0x00007f53cbf07eb3 in epoll_wait () from /lib64/libc.so.6
4 Thread 0x7f52b17fa700 (LWP 6429) “work_0” 0x00007f53cd01fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 0x7f52b87c9700 (LWP 6430) “daemon_6256_dx1” 0x00007f53cd01fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2 Thread 0x7f52b87a8700 (LWP 6565) “sphTimer” 0x00007f53cbefe9a3 in select () from /lib64/libc.so.6
Thread 11 (Thread 0x7f53cdb10700 (LWP 6256)):
#0 0x00007f53cd01fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000093f9d4 in ?? ()
#2 0x00007f53cdb0fc5f in ?? ()
#3 0x0000000001781d30 in ?? ()
#4 0x0000000001781d98 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x7f53cdaef700 (LWP 6257)):
#0 0x00007f53cbefe9a3 in select () from /lib64/libc.so.6
#1 0x00000000006704f4 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x000000000006e79d in ?? ()
#4 0x00000000000000f5 in ?? ()
#5 0x0000000000723fb0 in ?? ()
#6 0x0000000000000000 in ?? ()
Thread 9 (Thread 0x7f53cdace700 (LWP 6258)):
#0 0x00007f53cd01fde2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000072a9e6 in ?? ()
#2 0x000000005f49409a in ?? ()
#3 0x0000000018eb3f1b in ?? ()
#4 0x000000000019d837 in ?? ()
#5 0x0000000000e6de00 in ?? ()
#6 0x000000000019d837 in ?? ()
#7 0x0000000064f49774 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7f52edd6f700 (LWP 6383)):
#0 0x00007f53cd01fde2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000072a896 in ?? ()
#2 0x000000005f493c54 in ?? ()
#3 0x0000000027b85df7 in ?? ()
#4 0x000000000177f2d0 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7f52b9f47700 (LWP 6396)):
#0 0x00007f53cbf07eb3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000060556a in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000001788680 in ?? ()
#4 0x00000000ffffffff in ?? ()
#5 0x0000000000000001 in ?? ()
#6 0x00007f52b9f46b48 in ?? ()
#7 0x0000000000664b41 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7f52b3fff700 (LWP 6427)):
#0 0x00007f53cd01fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52b9ba81b4 in galera::ServiceThd::thd_func(void*) () from /lib/libgalera_manticore.so.31
#2 0x00007f53cd01bea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f53cbf078dd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f52b1ffb700 (LWP 6428)):
#0 0x00007f53cbf07eb3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f52b9bb9aa2 in asio::detail::epoll_reactor::run(bool, asio::detail::op_queueasio::detail::task_io_service_operation&) () from /lib/libgalera_manticore.so.31
#2 0x00007f52b9bbc01c in asio::detail::task_io_service::run(std::error_code&) () from /lib/libgalera_manticore.so.31
#3 0x00007f52b9be3e7b in gcomm::AsioProtonet::event_loop(gu::datetime::Period const&) () from /lib/libgalera_manticore.so.31
#4 0x00007f52b9b7967c in GCommConn::run() () from /lib/libgalera_manticore.so.31
#5 0x00007f52b9b80e89 in GCommConn::run_fn(void*) () from /lib/libgalera_manticore.so.31
#6 0x00007f53cd01bea5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f53cbf078dd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f52b17fa700 (LWP 6429)):
#0 0x00007f53cd01fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52b9b7aca9 in gcomm_recv(gcs_backend*, gcs_recv_msg*, long long) () from /lib/libgalera_manticore.so.31
#2 0x00007f52b9b6e592 in gcs_core_recv(gcs_core*, gcs_act_rcvd*, long long) () from /lib/libgalera_manticore.so.31
#3 0x00007f52b9b6940b in gcs_recv_thread(void*) () from /lib/libgalera_manticore.so.31
#4 0x00007f53cd01bea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f53cbf078dd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f52b87c9700 (LWP 6430)):
#0 0x00007f53cd01fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52b9cc8f5e in gu_fifo_get_head () from /lib/libgalera_manticore.so.31
#2 0x00007f52b9b67d57 in gcs_recv(gcs_conn*, gcs_action*) () from /lib/libgalera_manticore.so.31
#3 0x00007f52b9baf66b in galera::GcsActionSource::process(void*, bool&) () from /lib/libgalera_manticore.so.31
#4 0x00007f52b9b54c7b in galera::ReplicatorSMM::async_recv(void*) () from /lib/libgalera_manticore.so.31
#5 0x00007f52b9b45c6d in galera_recv () from /lib/libgalera_manticore.so.31
#6 0x000000000061840d in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f52b87a8700 (LWP 6565)):
#0 0x00007f53cbefe9a3 in select () from /lib64/libc.so.6
#1 0x00000000006704f4 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f53cdb12900 (LWP 6255)):
#0 0x00007f53cbefe9a3 in select () from /lib64/libc.so.6
#1 0x00000000006704f4 in ?? ()
#2 0x0000000000000000 in ?? ()
Main thread:
#0 0x00007f53cbefe9a3 in select () from /lib64/libc.so.6
#1 0x00000000006704f4 in ?? ()
#2 0x0000000000000000 in ?? ()
Local variables:
No symbol table info available.
[Inferior 1 (process 6255) detached]
— active threads —
thd 0 (work_1), proto mysql, state query, command update
— Totally 3 threads, and 1 client-working threads —
------- CRASH DUMP END -------
[root@wdebug-manticore1 manticore]# indextool --check dx1parts
Manticore 3.5.0 1d34c49@200722 release
Copyright (c) 2001-2016, Andrew Aksyonoff
using config file ‘/etc/manticoresearch/manticore.conf’…
checking index ‘dx1parts’…
checking schema…
check passed, 0.0 sec elapsed
BTW I found the query timeout setting that defaults to 5min
client_timeout
Maximum time to wait between requests (in seconds or special_suffixes) when using persistent connections. Optional, default is five minutes.
I opened a githb ticket and I now have an index (actually the entire manticore folder) that I can share with you. I uploaded the content of manticore folder and manticore.conf to your ftp server under github-issue-403 folder.