manticore-backup Failed to send query to the Manticore Search daemon

General Discussions - Russian
1

Доброго времени суток!

Имеется сервер с:

manticore-backup/jammy,now 1.3.8-24052208-57fc406 all [установлен, автоматически]
manticore-buddy/jammy,now 2.3.10-24052208-7612a4f all [установлен, автоматически]
manticore-columnar-lib/jammy,now 2.3.0-24052206-88a01c3 amd64 [установлен]
manticore-common/jammy,now 6.3.0-24052209-1811a9efb all [установлен, автоматически]
manticore-dev/jammy,now 6.3.0-24052209-1811a9efb all [установлен, автоматически]
manticore-executor/jammy,now 1.1.6-24052206-c55bc2b amd64 [установлен]
manticore-galera/jammy,now 3.37 amd64 [установлен]
manticore-icudata-65l/jammy,now 5.0.3-221123-d2d9e5e56 all [установлен, автоматически]
manticore-repo/now 0.0.4 all [установлен, локальный]
manticore-server-core/jammy,now 6.3.0-24052209-1811a9efb amd64 [установлен, автоматически]
manticore-server/jammy,now 6.3.0-24052209-1811a9efb amd64 [установлен, автоматически]
manticore-tools/jammy,now 6.3.0-24052209-1811a9efb amd64 [установлен, автоматически]
manticore-tzdata/jammy,now 1.0.0-240522-a8aa66e all [установлен, автоматически]
manticore/jammy,now 6.3.0-24052209-1811a9efb amd64 [установлен]

При выполнении команды manticore-backup:

manticore-backup --config=/etc/manticoresearch/manticore.conf --backup-dir=/mnt/backup --compress

в логе возникает ошибка:

Copyright (c) 2023-2024, Manticore Software LTD (https://manticoresearch.com)

Manticore config file: /etc/manticoresearch/manticore.conf
Tables to backup: all tables
Backup dir: /mnt/backup

Manticore config
  endpoint =  http://10.9.2.99:9312

Manticore versions:
  manticore: 6.3.0 1811a9efb@24052209
  columnar: 2.3.0 88a01c3@24052206
  secondary: 2.3.0 88a01c3@24052206
  knn: 2.3.0 88a01c3@24052206
  buddy: 2.3.10
2024-06-05 06:31:52 [Info] Starting the backup...
2024-06-05 06:32:06 [Info] Backing up config files...
2024-06-05 06:32:37 [Info]   config files - OK
2024-06-05 06:32:37 [Info] Backing up global state files...
2024-06-05 06:32:37 [Info]   global state files – OK
2024-06-05 06:32:41 [Info] Backing up tables...
2024-06-05 06:32:41 [Info]   bindings (rt) [1.52K]...
2024-06-05 06:32:52 [Info]    OK
2024-06-05 06:32:52 [Info]   categories (rt) [68.683K]...
2024-06-05 06:32:53 [Info]    OK
2024-06-05 06:32:53 [Info]   ekns (rt) [820B]...
2024-06-05 06:33:06 [Info]    OK
2024-06-05 06:33:06 [Info]   literature_work_cycle_volumes (rt) [506.537K]...
2024-06-05 06:33:29 [Info]    OK
2024-06-05 06:33:29 [Info]   literature_work_cycles (rt) [3.749M]...
2024-06-05 06:33:31 [Info]    OK
2024-06-05 06:33:31 [Info]   persons (rt) [29.019M]...
2024-06-05 06:33:32 [Info]    OK
2024-06-05 06:33:32 [Info]   product_collections (rt) [212.906K]...
2024-06-05 06:33:33 [Info]    OK
2024-06-05 06:33:33 [Info]   product_properties_values (rt) [866.989K]...
2024-06-05 06:33:33 [Info]    OK
2024-06-05 06:33:33 [Info]   product_tags (rt) [61.813K]...
2024-06-05 06:33:34 [Info]    OK
2024-06-05 06:33:34 [Info]   products (rt) [2.47G]...
2024-06-05 06:36:07 [Info]    OK
2024-06-05 06:36:07 [Info]   products_for_autocorrect (rt) [288.336M]...
2024-06-05 06:36:46 [Info]    OK
2024-06-05 06:36:46 [Info]   publisher_brands (rt) [220.358K]...
2024-06-05 06:36:47 [Info]    OK
2024-06-05 06:36:47 [Info]   publisher_series (rt) [9.315M]...
2024-06-05 06:36:48 [Info]    OK
2024-06-05 06:36:48 [Info]   publishers (rt) [1.151M]...
2024-06-05 06:36:51 [Info]    OK
2024-06-05 06:36:51 [Info]   retail_products (rt) [36.267G]...
2024-06-05 07:08:34 [Info]    OK
2024-06-05 07:08:34 [Info]   school_properties_values (rt) [258.705K]...
2024-06-05 07:08:53 [Info]    OK
2024-06-05 07:08:53 [Info]   shops (rt) [77.063K]...
2024-06-05 07:09:17 [Info]    OK
2024-06-05 07:09:17 [Info]   tbks (rt) [434.906K]...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info] Running sync
2024-06-05 07:09:47 [Info]  OK
2024-06-05 07:09:47 [Info] You can find backup here: /mnt/backup/backup-20240605063152
2024-06-05 07:09:47 [Info] Elapsed time: 2275.05s
2024-06-05 07:09:47 [Info] Done
2024-06-05 07:09:47 [Info] 
Unfreezing all tables...
2024-06-05 07:09:47 [Info]   bindings...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   categories...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   ekns...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   literature_work_cycle_volumes...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   literature_work_cycles...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   persons...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   product_collections...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   product_properties_values...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   product_tags...
2024-06-05 07:09:47 [Info]    OK
2024-06-05 07:09:47 [Info]   products...

Fatal error: Uncaught Manticoresearch\Backup\Exception\SearchdException: Failed to send query to the Manticore Search daemon. Ensure that it is set up to listen for HTTP or HTTPS connections and has the appropriate certificates in place. Additionally, check the 'max_connections' setting in
 the configuration file to ensure that it has not been exceeded.  in /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php:27
9
Stack trace:
#0 /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php(145): Manticoresearch\Backup\Lib\ManticoreClient->execute('UNFREEZE 
produc...')
#1 /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php(160): Manticoresearch\Backup\Lib\ManticoreClient->unfreeze(Array)
#2 [internal function]: Manticoresearch\Backup\Lib\ManticoreClient->Manticoresearch\Backup\Lib\{closure}(true, 'products')
#3 /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php(157): array_reduce(Array, Object(Closure), true)
#4 /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreBackup.php(166): Manticoresearch\Backup\Lib\ManticoreClient->unfreezeAll()
#5 [internal function]: Manticoresearch\Backup\Lib\ManticoreBackup::Manticoresearch\Backup\Lib\{closure}(Object(Manticoresearch\Backup\Lib\Mantic
oreClient))
#6 {main}
  thrown in /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php on line 279

Как видно сам backup при этом прошел, а на этапе UNFREEZE пошло что-то не так.
Это происходит периодически.
При этом сама Manticore повисает и более принимает запросов, кроме локального подключения.
Решается путем:

systemctl stop manticore
systemctl start manticore

В вот лог searchd:

[Wed Jun  5 06:24:00.115 2024] [3060672] rt: table retail_products: diskchunk 50508(83), segments 1  saved in 2.774805 (2.775184) sec, RAM saved/new 33993065/0 ratio 0.900000 (soft limit 120795955, conf limit 134217728)
[Wed Jun  5 06:24:04.316 2024] [3060673] rt: table retail_products: diskchunk 50507(84), segments 5  saved in 13.336653 (13.385911) sec, RAM saved/new 158546614/31004605 ratio 0.836432 (soft limit 112263938, conf limit 134217728)
[Wed Jun  5 06:24:05.555 2024] [3060672] rt: table retail_products: diskchunk 50509(85), segments 1  saved in 2.599324 (2.599727) sec, RAM saved/new 31004605/0 ratio 0.786432 (soft limit 105553051, conf limit 134217728)
[Wed Jun  5 06:24:33.045 2024] [3060673] rt: table retail_products: diskchunk 50512(83), segments 3  saved in 9.493652 (9.494065) sec, RAM saved/new 111616578/43061952 ratio 0.721604 (soft limit 96851990, conf limit 134217728)
[Wed Jun  5 06:24:35.781 2024] [3060671] rt: table retail_products: diskchunk 50513(84), segments 2  saved in 3.717802 (3.718218) sec, RAM saved/new 43061952/0 ratio 0.671604 (soft limit 90141104, conf limit 134217728)
[Wed Jun  5 06:25:00.201 2024] [3060670] rt: table retail_products: diskchunk 50515(85), segments 3  saved in 8.604319 (8.604775) sec, RAM saved/new 101979195/30010024 ratio 0.772633 (soft limit 103701014, conf limit 134217728)
[Wed Jun  5 06:25:23.237 2024] [3060672] rt: table retail_products: diskchunk 50517(84), segments 5  saved in 9.112808 (9.113299) sec, RAM saved/new 106337048/9306 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Wed Jun  5 06:25:30.767 2024] [3060672] WARNING: got ident_all
[Wed Jun  5 06:25:30.870 2024] [3060671] WARNING: rt common merge: table retail_products: failed to merge /var/lib/manticore/retail_products/retail_products.50512 (interrupted because of shutdown)
[Wed Jun  5 06:25:30.870 2024] [3060670] rt: table retail_products: optimized progressive chunk(s) 170 ( left 84 ) in 47m 55.8s
[Wed Jun  5 06:25:30.938 2024] [3060673] rt: table retail_products: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.009 Mb, time delta=7 sec, took=0.068 sec)
[Wed Jun  5 06:25:33.952 2024] [3060674] rt: table products: diskchunk 104352(13), segments 15  saved in 5.550124 (5.550515) sec, RAM saved/new 133089609/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Wed Jun  5 06:25:45.033 2024] [3060671] rt: table products: optimized progressive chunk(s) 13 ( left 12 ) in 11s 78ms
[Wed Jun  5 06:27:36.865 2024] [3060671] rt: table products: diskchunk 104354(13), segments 13  saved in 5.114076 (5.152566) sec, RAM saved/new 134260091/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Wed Jun  5 06:27:44.799 2024] [3060670] rt: table products: optimized progressive chunk(s) 13 ( left 12 ) in 7s 930ms
[Wed Jun  5 06:29:45.454 2024] [3060670] rt: table products: diskchunk 104356(13), segments 13  saved in 5.459832 (5.460300) sec, RAM saved/new 136490137/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Wed Jun  5 06:29:57.752 2024] [3060670] rt: table products: optimized progressive chunk(s) 13 ( left 12 ) in 12s 294ms
[Wed Jun  5 06:31:51.941 2024] [3060673] WARNING: got ident_all
[Wed Jun  5 06:31:54.358 2024] [3060671] rt: table products: diskchunk 104358(13), segments 13  saved in 5.617763 (5.618163) sec, RAM saved/new 134159092/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Wed Jun  5 06:32:06.360 2024] [3060672] rt: table products: optimized progressive chunk(s) 13 ( left 12 ) in 11s 999ms
[Wed Jun  5 06:32:37.823 2024] [3060671] WARNING: got ident_all
[Wed Jun  5 06:32:37.827 2024] [3060671] last message repeated 17 times
[Wed Jun  5 06:32:37.827 2024] [3060671] rt: table bindings: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.001 Mb, time delta=3431 sec, took=0.003 sec)
[Wed Jun  5 06:32:37.828 2024] [3060673] rt: table categories: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.075 Mb, time delta=3431 sec, took=0.000 sec)
[Wed Jun  5 06:32:37.831 2024] [3060673] rt: table ekns: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.000 Mb, time delta=3429 sec, took=0.002 sec)
[Wed Jun  5 06:32:37.846 2024] [3060674] rt: table literature_work_cycle_volumes: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.562 Mb, time delta=3431 sec, took=0.015 sec)
[Wed Jun  5 06:32:37.923 2024] [3060670] rt: table literature_work_cycles: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=3.016 Mb, time delta=3431 sec, took=0.077 sec)
[Wed Jun  5 06:32:38.190 2024] [3060673] rt: table persons: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=31.319 Mb, time delta=3431 sec, took=0.266 sec)
[Wed Jun  5 06:32:38.195 2024] [3060672] rt: table product_collections: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.226 Mb, time delta=3431 sec, took=0.005 sec)
[Wed Jun  5 06:32:38.247 2024] [3060672] rt: table product_properties_values: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.984 Mb, time delta=3431 sec, took=0.052 sec)
[Wed Jun  5 06:32:38.252 2024] [3060672] rt: table product_tags: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.072 Mb, time delta=3432 sec, took=0.003 sec)
[Wed Jun  5 06:32:38.303 2024] [3060672] rt: table products: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=49.014 Mb, time delta=43 sec, took=0.050 sec)
[Wed Jun  5 06:32:40.820 2024] [3060671] rt: table products_for_autocorrect: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=114.741 Mb, time delta=3433 sec, took=2.507 sec)
[Wed Jun  5 06:32:40.843 2024] [3060671] rt: table publisher_brands: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.263 Mb, time delta=3434 sec, took=0.007 sec)
[Wed Jun  5 06:32:40.959 2024] [3060671] rt: table publisher_series: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=10.670 Mb, time delta=3434 sec, took=0.115 sec)
[Wed Jun  5 06:32:40.987 2024] [3060674] rt: table publishers: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=1.327 Mb, time delta=3434 sec, took=0.027 sec)
[Wed Jun  5 06:32:40.997 2024] [3060674] rt: table retail_products: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.009 Mb, time delta=430 sec, took=0.010 sec)
[Wed Jun  5 06:32:41.010 2024] [3060674] rt: table school_properties_values: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.285 Mb, time delta=3434 sec, took=0.007 sec)
[Wed Jun  5 06:32:41.015 2024] [3060674] rt: table shops: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.083 Mb, time delta=3433 sec, took=0.003 sec)
[Wed Jun  5 06:32:41.016 2024] [3060674] rt: table tbks: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=0.475 Mb, time delta=3434 sec, took=0.001 sec)
[Wed Jun  5 06:32:41.027 2024] [3060670] WARNING: got ident_all
[Wed Jun  5 06:32:52.853 2024] [3060672] WARNING: got ident_all
[Wed Jun  5 06:32:55.124 2024] [3060669] WARNING: last message repeated 3 times
[Wed Jun  5 06:33:06.826 2024] [3060673] WARNING: got ident_all
[Wed Jun  5 06:33:08.138 2024] [3060669] WARNING: last message repeated 1 times
[Wed Jun  5 06:33:29.933 2024] [3060671] WARNING: got ident_all
[Wed Jun  5 06:33:31.080 2024] [3060672] WARNING: last message repeated 1 times
[Wed Jun  5 06:33:31.080 2024] [3060672] WARNING: got ident_all
[Wed Jun  5 06:33:32.165 2024] [3060669] WARNING: last message repeated 1 times
[Wed Jun  5 06:33:32.504 2024] [3060674] WARNING: got ident_all
[Wed Jun  5 06:33:35.669 2024] [3060669] WARNING: last message repeated 7 times
[Wed Jun  5 06:36:07.926 2024] [3060670] WARNING: got ident_all
[Wed Jun  5 06:36:09.337 2024] [3060669] WARNING: last message repeated 1 times
[Wed Jun  5 06:36:46.719 2024] [3060674] WARNING: got ident_all
[Wed Jun  5 06:36:49.382 2024] [3060669] WARNING: last message repeated 5 times
[Wed Jun  5 06:36:51.291 2024] [3060674] WARNING: got ident_all
[Wed Jun  5 06:36:52.386 2024] [3060669] WARNING: last message repeated 1 times
[Wed Jun  5 07:08:34.768 2024] [3060673] WARNING: got ident_all
[Wed Jun  5 07:08:35.976 2024] [3060669] WARNING: last message repeated 1 times
[Wed Jun  5 07:08:53.386 2024] [3060673] WARNING: got ident_all
[Wed Jun  5 07:08:54.496 2024] [3060669] WARNING: last message repeated 1 times
[Wed Jun  5 07:09:17.988 2024] [3060672] WARNING: got ident_all
[Wed Jun  5 07:09:19.022 2024] [3060669] WARNING: last message repeated 1 times
[Wed Jun  5 07:09:47.364 2024] [3060671] WARNING: got ident_all
[Wed Jun  5 07:09:49.056 2024] [3060669] WARNING: last message repeated 10 times
[Wed Jun  5 07:23:23.759 2024] [3060669] caught SIGTERM, shutting down
[Wed Jun  5 07:23:23.760 2024] [3060672] WARNING: [BUDDY] [19010] error: Interrupted due to shutdown
[Wed Jun  5 07:27:28.616 2024] [3060668] watchdog: main process 3060669 killed cleanly with SIGKILL, shutting down
[Wed Jun  5 07:27:37.061 2024] [3074640] watchdog: main process 3074641 forked ok
[Wed Jun  5 07:27:37.070 2024] [3074641] Using local time zone '/etc/localtime'
[Wed Jun  5 07:27:37.071 2024] [3074641] starting daemon version '6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
[Wed Jun  5 07:27:37.071 2024] [3074641] listening on 10.9.2.99:9312 for sphinx and http(s)
[Wed Jun  5 07:27:37.071 2024] [3074641] listening on 10.9.2.99:9306 for mysql
[Wed Jun  5 07:27:37.071 2024] [3074641] listening on 10.9.2.99:9308 for sphinx and http(s)
[Wed Jun  5 07:27:37.220 2024] [3074642] WARNING: table 'publisher_brands': table 'publisher_brands': morphology option changed from config has no effect, ignoring
[Wed Jun  5 07:27:37.236 2024] [3074645] WARNING: table 'publishers': table 'publishers': morphology option changed from config has no effect, ignoring
[Wed Jun  5 07:27:37.330 2024] [3074644] WARNING: table 'publisher_series': table 'publisher_series': morphology option changed from config has no effect, ignoring
[Wed Jun  5 07:27:37.331 2024] [3074642] WARNING: table 'school_properties_values': table 'school_properties_values': morphology option changed from config has no effect, ignoring
[Wed Jun  5 07:27:37.505 2024] [3074643] WARNING: table 'persons': table 'persons': morphology option changed from config has no effect, ignoring
[Wed Jun  5 07:27:37.519 2024] [3074645] WARNING: table 'product_properties_values': table 'product_properties_values': morphology option changed from config has no effect, ignoring
[Wed Jun  5 07:27:37.968 2024] [3074643] WARNING: table 'products': table 'products': morphology option changed from config has no effect, ignoring
[Wed Jun  5 07:27:37.975 2024] [3074643] WARNING: missing /var/lib/manticore/products_for_autocorrect/products_for_autocorrect.1032.spidx; secondary index(es) disabled, consider using ALTER REBUILD SECONDARY to recover the secondary index
[Wed Jun  5 07:27:37.978 2024] [3074643] WARNING: missing /var/lib/manticore/products_for_autocorrect/products_for_autocorrect.1082.spidx; secondary index(es) disabled, consider using ALTER REBUILD SECONDARY to recover the secondary index
[Wed Jun  5 07:27:37.981 2024] [3074643] WARNING: missing /var/lib/manticore/products_for_autocorrect/products_for_autocorrect.904.spidx; secondary index(es) disabled, consider using ALTER REBUILD SECONDARY to recover the secondary index
[Wed Jun  5 07:27:37.985 2024] [3074643] WARNING: missing /var/lib/manticore/products_for_autocorrect/products_for_autocorrect.822.spidx; secondary index(es) disabled, consider using ALTER REBUILD SECONDARY to recover the secondary index
[Wed Jun  5 07:27:37.995 2024] [3074643] WARNING: missing /var/lib/manticore/products_for_autocorrect/products_for_autocorrect.932.spidx; secondary index(es) disabled, consider using ALTER REBUILD SECONDARY to recover the secondary index
[Wed Jun  5 07:27:38.000 2024] [3074643] WARNING: missing /var/lib/manticore/products_for_autocorrect/products_for_autocorrect.928.spidx; secondary index(es) disabled, consider using ALTER REBUILD SECONDARY to recover the secondary index
[Wed Jun  5 07:27:38.005 2024] [3074643] WARNING: missing /var/lib/manticore/products_for_autocorrect/products_for_autocorrect.1021.spidx; secondary index(es) disabled, consider using ALTER REBUILD SECONDARY to recover the secondary index
[Wed Jun  5 07:27:39.977 2024] [3074645] prereading 18 tables
[Wed Jun  5 07:27:39.987 2024] [3074641] accepting connections
[Wed Jun  5 07:27:40.072 2024] [3074643] [BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://10.9.2.99:9312 --bind=127.0.0.1 --disable-telemetry --threads=5 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:34621
[Wed Jun  5 07:27:40.087 2024] [3074642] [BUDDY] Loaded plugins:
[Wed Jun  5 07:27:40.087 2024] [3074642] [BUDDY]   core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
[Wed Jun  5 07:27:40.087 2024] [3074642] [BUDDY]   local: 
[Wed Jun  5 07:27:40.087 2024] [3074642] [BUDDY]   extra: 
[Wed Jun  5 07:28:19.136 2024] [3074643] rt: table retail_products: diskchunk 50520(85), segments 1  saved in 2.241569 (2.253052) sec, RAM saved/new 25837171/0 ratio 0.900000 (soft limit 120795955, conf limit 134217728)
[Wed Jun  5 07:30:50.336 2024] [3074645] preread 18 tables in 190.359 sec

Вот здесь был перезапуск, чтобы восстановить работу:

[Wed Jun  5 07:23:23.759 2024] [3060669] caught SIGTERM, shutting down
[Wed Jun  5 07:23:23.760 2024] [3060672] WARNING: [BUDDY] [19010] error: Interrupted due to shutdown
[Wed Jun  5 07:27:28.616 2024] [3060668] watchdog: main process 3060669 killed cleanly with SIGKILL, shutting down
[Wed Jun  5 07:27:37.061 2024] [3074640] watchdog: main process 3074641 forked ok

Подскажите, пожалуйста, с чем связано такое поведение и как устранить?

2

мне кажется что тут лучше завести тикет на Github и в рамках этого тикета вести расследование

3

Тут GitHub - manticoresoftware/manticoresearch-backup: Repository for Manticore Search backup scripts ?

4

да можно и там

1 Like
5

можете попробовать еще сделать

unfreeze table_name

на таблицы которые использовались - чтобы убедится, что это поможет без перезапуска демона?

6

Я так полагаю последней таблице, где ошибка и возникла и потом по остальным пройти?
Попробую как снова возникнет такая же ошибка.

7

можно начать с последней, где ошибка случилась и пробовать - восстановилась ли отзывчивость демона

1 Like
8

Проверил этот способ, но не помог.
Ошибка такая же и так же на этом же index products:

Unfreezing all tables...
2024-06-05 08:50:53 [Info]   bindings...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   categories...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   ekns...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   literature_work_cycle_volumes...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   literature_work_cycles...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   persons...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   product_collections...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   product_properties_values...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   product_tags...
2024-06-05 08:50:53 [Info]    OK
2024-06-05 08:50:53 [Info]   products...

Fatal error: Uncaught Manticoresearch\Backup\Exception\SearchdException: Failed to send query to the Manticore Search daemon. Ensure that it is set up to listen for HTTP or HTTPS connections and has the appropriate certificates in place. Additionally, check the 'max_connections' setting in the configuration file to ensure that it has not been exceeded.  in /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php:279
Stack trace:
#0 /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php(145): Manticoresearch\Backup\Lib\ManticoreClient->execute('UNFREEZE produc...')
#1 /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php(160): Manticoresearch\Backup\Lib\ManticoreClient->unfreeze(Array)
#2 [internal function]: Manticoresearch\Backup\Lib\ManticoreClient->Manticoresearch\Backup\Lib\{closure}(true, 'products')
#3 /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php(157): array_reduce(Array, Object(Closure), true)
#4 /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreBackup.php(166): Manticoresearch\Backup\Lib\ManticoreClient->unfreezeAll()
#5 [internal function]: Manticoresearch\Backup\Lib\ManticoreBackup::Manticoresearch\Backup\Lib\{closure}(Object(Manticoresearch\Backup\Lib\ManticoreClient))
#6 {main}
  thrown in /usr/share/manticore/modules/manticore-backup/src/Lib/ManticoreClient.php on line 279

Запустил команду:

 UNFREEZE products;

но она просто повисла.
Выполнил запрос:

select * from @@system.sessions limit 10000\G

результат дал 80 записей, из которых есть 2:

*************************** 70. row ***************************
           id: 71
        proto: http
        state: query
         host: 10.9.2.99:42510
       connid: 20532
       killed: 0
last cmd time: 6m ago
     last cmd: unfreeze_indexes
*************************** 71. row ***************************
           id: 72
        proto: mysql
        state: query
         host: 10.9.2.99:40886
       connid: 20645
       killed: 0
last cmd time: 3m ago
     last cmd: UNFREEZE products

Пришлось перезапустить демона.

9

И после перезапуска добавились новые ошибки:

[Wed Jun  5 08:58:29.395 2024] [3074641] caught SIGTERM, shutting down
[Wed Jun  5 09:02:06.113 2024] [3086345] watchdog: main process 3086346 forked ok
[Wed Jun  5 09:02:06.123 2024] [3086346] Using local time zone '/etc/localtime'
[Wed Jun  5 09:02:06.124 2024] [3086346] starting daemon version '6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
[Wed Jun  5 09:02:06.124 2024] [3086346] listening on 10.9.2.99:9312 for sphinx and http(s)
[Wed Jun  5 09:02:06.124 2024] [3086346] listening on 10.9.2.99:9306 for mysql
[Wed Jun  5 09:02:06.124 2024] [3086346] listening on 10.9.2.99:9308 for sphinx and http(s)
[Wed Jun  5 09:02:06.148 2024] [3086347] WARNING: table 'literature_work_cycles': disabled at the JSON config
[Wed Jun  5 09:02:06.148 2024] [3086347] WARNING: table 'literature_work_cycles': prealloc: failed to lock /var/lib/manticore/literature_work_cycles/literature_work_cycles.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.150 2024] [3086347] WARNING: table 'publisher_brands': disabled at the JSON config
[Wed Jun  5 09:02:06.150 2024] [3086347] WARNING: table 'publisher_brands': prealloc: failed to lock /var/lib/manticore/publisher_brands/publisher_brands.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.151 2024] [3086347] WARNING: table 'publishers': disabled at the JSON config
[Wed Jun  5 09:02:06.151 2024] [3086347] WARNING: table 'publishers': prealloc: failed to lock /var/lib/manticore/publishers/publishers.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.151 2024] [3086347] WARNING: table 'tbks': disabled at the JSON config
[Wed Jun  5 09:02:06.151 2024] [3086347] WARNING: table 'tbks': prealloc: failed to lock /var/lib/manticore/tbks/tbks.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.152 2024] [3086347] WARNING: table 'product_tags': disabled at the JSON config
[Wed Jun  5 09:02:06.152 2024] [3086347] WARNING: table 'product_tags': prealloc: failed to lock /var/lib/manticore/product_tags/product_tags.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.152 2024] [3086347] WARNING: table 'publisher_series': disabled at the JSON config
[Wed Jun  5 09:02:06.152 2024] [3086347] WARNING: table 'publisher_series': prealloc: failed to lock /var/lib/manticore/publisher_series/publisher_series.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.153 2024] [3086347] WARNING: table 'school_properties_values': disabled at the JSON config
[Wed Jun  5 09:02:06.153 2024] [3086347] WARNING: table 'school_properties_values': prealloc: failed to lock /var/lib/manticore/school_properties_values/school_properties_values.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.153 2024] [3086347] WARNING: table 'categories': disabled at the JSON config
[Wed Jun  5 09:02:06.153 2024] [3086347] WARNING: table 'categories': prealloc: failed to lock /var/lib/manticore/categories/categories.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.154 2024] [3086347] WARNING: table 'literature_work_cycle_volumes': disabled at the JSON config
[Wed Jun  5 09:02:06.154 2024] [3086347] WARNING: table 'literature_work_cycle_volumes': prealloc: failed to lock /var/lib/manticore/literature_work_cycle_volumes/literature_work_cycle_volumes.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.154 2024] [3086347] WARNING: table 'persons': disabled at the JSON config
[Wed Jun  5 09:02:06.154 2024] [3086347] WARNING: table 'persons': prealloc: failed to lock /var/lib/manticore/persons/persons.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.155 2024] [3086347] WARNING: table 'bindings': disabled at the JSON config
[Wed Jun  5 09:02:06.155 2024] [3086347] WARNING: table 'bindings': prealloc: failed to lock /var/lib/manticore/bindings/bindings.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.155 2024] [3086347] WARNING: table 'product_collections': disabled at the JSON config
[Wed Jun  5 09:02:06.155 2024] [3086347] WARNING: table 'product_collections': prealloc: failed to lock /var/lib/manticore/product_collections/product_collections.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.155 2024] [3086347] WARNING: table 'product_properties_values': disabled at the JSON config
[Wed Jun  5 09:02:06.155 2024] [3086347] WARNING: table 'product_properties_values': prealloc: failed to lock /var/lib/manticore/product_properties_values/product_properties_values.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.156 2024] [3086347] WARNING: table 'products': disabled at the JSON config
[Wed Jun  5 09:02:06.156 2024] [3086347] WARNING: table 'products': prealloc: failed to lock /var/lib/manticore/products/products.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.157 2024] [3086347] WARNING: table 'products_for_autocorrect': disabled at the JSON config
[Wed Jun  5 09:02:06.157 2024] [3086347] WARNING: table 'products_for_autocorrect': prealloc: failed to lock /var/lib/manticore/products_for_autocorrect/products_for_autocorrect.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.158 2024] [3086347] WARNING: table 'shops': disabled at the JSON config
[Wed Jun  5 09:02:06.158 2024] [3086347] WARNING: table 'shops': prealloc: failed to lock /var/lib/manticore/shops/shops.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.159 2024] [3086347] WARNING: table 'retail_products': disabled at the JSON config
[Wed Jun  5 09:02:06.159 2024] [3086347] WARNING: table 'retail_products': prealloc: failed to lock /var/lib/manticore/retail_products/retail_products.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.160 2024] [3086347] WARNING: table 'ekns': disabled at the JSON config
[Wed Jun  5 09:02:06.160 2024] [3086347] WARNING: table 'ekns': prealloc: failed to lock /var/lib/manticore/ekns/ekns.lock: Resource temporarily unavailable - NOT SERVING
[Wed Jun  5 09:02:06.160 2024] [3086349] prereading 0 tables
[Wed Jun  5 09:02:06.160 2024] [3086349] preread 0 tables in 0.000 sec
[Wed Jun  5 09:02:06.165 2024] [3086346] accepting connections
[Wed Jun  5 09:02:09.168 2024] [3086351] WARNING: [BUDDY] failed to start after 3 sec

еще один перезапуск и пока живет.

10

Завел тикет на Github: manticore-backup Failed to send query to the Manticore Search daemon · Issue #116 · manticoresoftware/manticoresearch-backup · GitHub