Manticore queries are very slow, taking over 60 seconds. But, then I reboot the manticore, running as a windows service, and then the queries get very fast again, under 5 seconds as expected. I tried adding the rt_flush_period parameter to the conf file, but that did not help – only rebooting the windows service helps. Is there any particular function that takes place during service reboot that can be done to keep the queries fast instead of rebooting?
Sounds like a bug. It could be a memory leak causing swapping, which then slows down the search. Can you check if that’s the case?
Your searchd log and query log would be also helpful.
Here is my searchd log. looks like there is some crash happening. i don’t have query log available, unfortunately. Still trying to determine any memory leak issues.
[Wed Feb 12 19:38:31.851 2025] [14180] [BUDDY] started v1.0.18 ‘“E:\Manticore\bin\manticore-executor\manticore-executor.exe” “E:\Manticore/share/modules/manticore-buddy\src\main.php” --listen=http://11.11.111.11:9306 --threads=14’ at http://127.0.0.1:54950
[Wed Feb 12 19:38:32.216 2025] [5988] [BUDDY] Loaded plugins:
[Wed Feb 12 19:38:32.216 2025] [5988] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Wed Feb 12 19:38:32.216 2025] [5988] [BUDDY] local:
[Wed Feb 12 19:38:32.216 2025] [5988] [BUDDY] extra:
[Wed Feb 12 20:37:05.401 2025] [11892] WARNING: conn 11.11.111.11:55283(68), sock=5728: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Wed Feb 12 20:37:05.514 2025] [6572] rt: table manticore_people_index: ramchunk saved in 0.112 sec
[Wed Feb 12 20:37:05.702 2025] [6572] rt: table manticore_case_index: ramchunk saved in 0.084 sec
[Wed Feb 12 20:37:05.841 2025] [6572] rt: table manticore_document_index: ramchunk saved in 0.074 sec
[Wed Feb 12 20:37:06.038 2025] [10436] shutdown daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Wed Feb 12 20:37:06.038 2025] [10436] shutdown complete
[Wed Feb 12 20:37:10.097 2025] [19204] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Wed Feb 12 20:37:10.105 2025] [19204] listening on 11.11.111.11:9306 for sphinx and http(s)
[Wed Feb 12 20:37:10.811 2025] [13720] prereading 3 tables
[Wed Feb 12 20:37:10.880 2025] [19204] accepting connections
[Wed Feb 12 20:37:11.180 2025] [13720] preread 3 tables in 0.369 sec
[Wed Feb 12 20:37:11.356 2025] [3984] [BUDDY] started v1.0.18 ‘“E:\Manticore\bin\manticore-executor\manticore-executor.exe” “E:\Manticore/share/modules/manticore-buddy\src\main.php” --listen=http://11.11.111.11:9306 --threads=14’ at http://127.0.0.1:55291
[Wed Feb 12 20:37:11.673 2025] [688] [BUDDY] Loaded plugins:
[Wed Feb 12 20:37:11.673 2025] [688] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Wed Feb 12 20:37:11.673 2025] [688] [BUDDY] local:
[Wed Feb 12 20:37:11.673 2025] [688] [BUDDY] extra:
[Thu Feb 13 14:49:03.419 2025] [18344] rt: table manticore_people_index: ramchunk saved in 0.112 sec
[Thu Feb 13 14:49:03.669 2025] [18344] rt: table manticore_case_index: ramchunk saved in 0.073 sec
[Thu Feb 13 14:49:03.780 2025] [18344] rt: table manticore_document_index: ramchunk saved in 0.060 sec
[Thu Feb 13 14:49:03.967 2025] [19204] shutdown daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Thu Feb 13 14:49:03.967 2025] [19204] shutdown complete
[Thu Feb 13 14:49:30.471 2025] [18120] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Thu Feb 13 14:49:30.480 2025] [18120] listening on 11.11.111.11:9306 for sphinx and http(s)
[Thu Feb 13 14:49:38.374 2025] [21468] prereading 3 tables
[Thu Feb 13 14:49:38.572 2025] [18120] accepting connections
[Thu Feb 13 14:49:38.752 2025] [21468] preread 3 tables in 0.378 sec
[Thu Feb 13 14:49:39.717 2025] [19356] [BUDDY] started v1.0.18 ‘“E:\Manticore\bin\manticore-executor\manticore-executor.exe” “E:\Manticore/share/modules/manticore-buddy\src\main.php” --listen=http://11.11.111.11:9306 --threads=14’ at http://127.0.0.1:55844
[Thu Feb 13 14:49:40.076 2025] [16668] [BUDDY] Loaded plugins:
[Thu Feb 13 14:49:40.076 2025] [16668] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Thu Feb 13 14:49:40.076 2025] [16668] [BUDDY] local:
[Thu Feb 13 14:49:40.076 2025] [16668] [BUDDY] extra:
[Thu Feb 13 16:04:53.299 2025] [3752] WARNING: conn 11.11.111.11:56346(111), sock=5900: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Thu Feb 13 16:04:53.415 2025] [15420] rt: table manticore_people_index: ramchunk saved in 0.115 sec
[Thu Feb 13 16:04:53.586 2025] [15420] rt: table manticore_case_index: ramchunk saved in 0.081 sec
[Thu Feb 13 16:04:53.720 2025] [15420] rt: table manticore_document_index: ramchunk saved in 0.073 sec
[Thu Feb 13 16:04:53.906 2025] [18120] shutdown daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Thu Feb 13 16:04:53.906 2025] [18120] shutdown complete
[Thu Feb 13 16:04:59.313 2025] [13460] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Thu Feb 13 16:04:59.322 2025] [13460] listening on 11.11.111.11:9306 for sphinx and http(s)
[Thu Feb 13 16:05:00.037 2025] [11328] prereading 3 tables
[Thu Feb 13 16:05:00.210 2025] [13460] accepting connections
[Thu Feb 13 16:05:00.515 2025] [11328] preread 3 tables in 0.479 sec
[Thu Feb 13 16:05:00.656 2025] [18836] [BUDDY] started v1.0.18 ‘“E:\Manticore\bin\manticore-executor\manticore-executor.exe” “E:\Manticore/share/modules/manticore-buddy\src\main.php” --listen=http://11.11.111.11:9306 --threads=14’ at http://127.0.0.1:56357
[Thu Feb 13 16:05:00.974 2025] [1772] [BUDDY] Loaded plugins:
[Thu Feb 13 16:05:00.974 2025] [1772] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Thu Feb 13 16:05:00.974 2025] [1772] [BUDDY] local:
[Thu Feb 13 16:05:00.974 2025] [1772] [BUDDY] extra:
[Thu Feb 13 16:32:26.644 2025] [860] WARNING: conn 11.11.111.11:56605(41), sock=5708: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Thu Feb 13 16:32:26.765 2025] [372] rt: table manticore_people_index: ramchunk saved in 0.120 sec
[Thu Feb 13 16:32:26.965 2025] [372] rt: table manticore_case_index: ramchunk saved in 0.091 sec
[Thu Feb 13 16:32:27.117 2025] [372] rt: table manticore_document_index: ramchunk saved in 0.083 sec
[Thu Feb 13 16:32:27.328 2025] [13460] shutdown daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Thu Feb 13 16:32:27.328 2025] [13460] shutdown complete
------- FATAL: CRASH DUMP -------
[Thu Feb 13 16:32:26.640 2025] [ 3252]
[Thu Feb 13 16:32:27.348 2025] [13460] WARNING: GlobalCrashQueryGetRef: thread-local info is not set! Use ad-hoc
— crashed invalid query —
— request dump end —
— local index:
minidump located at: E:\Manticore/var/run/manticore/searchd.pid.3252.0000000000000000.mdmp
— active threads —
— Totally 0 threads, and 0 client-working threads —
------- CRASH DUMP END -------
[Thu Feb 13 16:32:29.966 2025] [15868] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Thu Feb 13 16:32:29.975 2025] [15868] listening on 11.11.111.11:9306 for sphinx and http(s)
[Thu Feb 13 16:32:30.678 2025] [19332] prereading 3 tables
[Thu Feb 13 16:32:30.763 2025] [15868] accepting connections
[Thu Feb 13 16:32:31.048 2025] [19332] preread 3 tables in 0.371 sec
[Thu Feb 13 16:32:31.271 2025] [4852] [BUDDY] started v1.0.18 ‘“E:\Manticore\bin\manticore-executor\manticore-executor.exe” “E:\Manticore/share/modules/manticore-buddy\src\main.php” --listen=http://11.11.111.11:9306 --threads=14’ at http://127.0.0.1:56613
[Thu Feb 13 16:32:31.615 2025] [13076] [BUDDY] Loaded plugins:
[Thu Feb 13 16:32:31.615 2025] [13076] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Thu Feb 13 16:32:31.615 2025] [13076] [BUDDY] local:
[Thu Feb 13 16:32:31.615 2025] [13076] [BUDDY] extra:
[Thu Feb 13 16:45:41.458 2025] [17268] WARNING: conn 11.11.111.11:56639(11), sock=5684: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Thu Feb 13 16:45:41.571 2025] [8564] rt: table manticore_people_index: ramchunk saved in 0.112 sec
[Thu Feb 13 16:45:41.741 2025] [13964] rt: table manticore_case_index: ramchunk saved in 0.084 sec
[Thu Feb 13 16:45:41.873 2025] [13964] rt: table manticore_document_index: ramchunk saved in 0.073 sec
[Thu Feb 13 16:45:42.075 2025] [15868] shutdown daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Thu Feb 13 16:45:42.076 2025] [15868] shutdown complete
[Thu Feb 13 16:45:50.866 2025] [16888] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Thu Feb 13 16:45:50.874 2025] [16888] listening on 11.11.111.11:9306 for sphinx and http(s)
[Thu Feb 13 16:45:51.563 2025] [20988] prereading 3 tables
[Thu Feb 13 16:45:51.655 2025] [16888] accepting connections
[Thu Feb 13 16:45:51.957 2025] [20988] preread 3 tables in 0.394 sec
[Thu Feb 13 16:45:52.145 2025] [1160] [BUDDY] started v1.0.18 ‘“E:\Manticore\bin\manticore-executor\manticore-executor.exe” “E:\Manticore/share/modules/manticore-buddy\src\main.php” --listen=http://11.11.111.11:9306 --threads=14’ at http://127.0.0.1:56692
[Thu Feb 13 16:45:52.460 2025] [11588] [BUDDY] Loaded plugins:
[Thu Feb 13 16:45:52.460 2025] [11588] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Thu Feb 13 16:45:52.460 2025] [11588] [BUDDY] local:
[Thu Feb 13 16:45:52.460 2025] [11588] [BUDDY] extra:
[Sat Feb 15 06:22:54.041 2025] [5700] rt: table manticore_people_index: ramchunk saved in 0.128 sec
[Sat Feb 15 06:22:54.220 2025] [5700] rt: table manticore_case_index: ramchunk saved in 0.086 sec
[Sat Feb 15 06:22:54.353 2025] [5700] rt: table manticore_document_index: ramchunk saved in 0.078 sec
[Sat Feb 15 06:22:54.519 2025] [16888] shutdown daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Sat Feb 15 06:22:54.519 2025] [16888] shutdown complete
[Sat Feb 15 06:23:49.619 2025] [5988] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Sat Feb 15 06:23:49.625 2025] [5988] listening on 11.11.111.11:9306 for sphinx and http(s)
[Sat Feb 15 06:24:14.332 2025] [6060] prereading 3 tables
[Sat Feb 15 06:24:14.856 2025] [5988] accepting connections
[Sat Feb 15 06:24:17.859 2025] [6072] WARNING: [BUDDY] failed to start after 3 sec
[Sat Feb 15 06:25:17.527 2025] [6060] preread 3 tables in 63.195 sec
[Tue Feb 25 15:57:05.652 2025] [6060] WARNING: conn 11.11.111.11:55963(486), sock=6076: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Tue Feb 25 15:57:05.811 2025] [6096] rt: table manticore_people_index: ramchunk saved in 0.096 sec
[Tue Feb 25 15:57:06.077 2025] [6096] rt: table manticore_case_index: ramchunk saved in 0.083 sec
[Tue Feb 25 15:57:06.206 2025] [6096] rt: table manticore_document_index: ramchunk saved in 0.060 sec
[Tue Feb 25 15:57:06.459 2025] [5988] shutdown daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Tue Feb 25 15:57:06.459 2025] [5988] shutdown complete
[Tue Feb 25 15:57:55.045 2025] [16584] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Tue Feb 25 15:57:55.055 2025] [16584] listening on 11.11.111.11:9306 for sphinx and http(s)
[Tue Feb 25 15:57:59.135 2025] [1192] prereading 3 tables
[Tue Feb 25 15:57:59.476 2025] [16584] accepting connections
[Tue Feb 25 15:57:59.648 2025] [1192] preread 3 tables in 0.512 sec
[Tue Feb 25 15:58:02.483 2025] [4084] WARNING: [BUDDY] failed to start after 3 sec
[Tue Feb 25 16:06:30.581 2025] [13740] WARNING: conn 11.11.111.11:56037(4), sock=5716: failed to receive HTTP request, AsyncNetInputBuffer_c::AppendData: got SIGTERM, return -1
[Tue Feb 25 16:06:30.711 2025] [16624] rt: table manticore_people_index: ramchunk saved in 0.129 sec
[Tue Feb 25 16:06:30.897 2025] [16624] rt: table manticore_case_index: ramchunk saved in 0.081 sec
[Tue Feb 25 16:06:31.069 2025] [16624] rt: table manticore_document_index: ramchunk saved in 0.088 sec
[Tue Feb 25 16:06:31.281 2025] [16584] shutdown daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Tue Feb 25 16:06:31.281 2025] [16584] shutdown complete
------- FATAL: CRASH DUMP -------
[Tue Feb 25 16:06:30.576 2025] [11192]
[Tue Feb 25 16:06:31.426 2025] [16584] WARNING: GlobalCrashQueryGetRef: thread-local info is not set! Use ad-hoc
— crashed invalid query —
— request dump end —
— local index:
minidump located at: E:\Manticore/var/run/manticore/searchd.pid.11192.0000000000000000.mdmp
— active threads —
— Totally 0 threads, and 0 client-working threads —
------- CRASH DUMP END -------
[Tue Feb 25 16:06:43.152 2025] [15232] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
[Tue Feb 25 16:06:43.160 2025] [15232] listening on 11.11.111.11:9306 for sphinx and http(s)
Many crashes and bugs have been fixed since version 6.2.12, which was released 1.5 years ago. We recommend upgrading to the latest release. Don’t forget to review the breaking changes in the changelog: Changelog | Manticore Search Manual
Will do, thank you!