I am trying to build a POC with a manticore multimaster instance & persistent volumes stored on a NFS instance.
For storage i am using nutanix csi.
I have downloaded the Helm chart in my Rancher ui and i am trying to deploy with 3 replica’s.
There is no replica deployed because of:
[2025-02-20T17:12:41.338130+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away.
I can get this running on a “normal” where i can edit manticore.conf , but in the k8s scenario i don’t see how i can do it.
please help, i’m stuck.
Please provide more logs. The error MySQL server has gone away could be due to a timeout or a crash. It’s hard to determine the cause without more details.
Thanks @Sergey .Here is the full log. I hope that this is the one you are looking for. If not let me know what else is needed and i will happily provide:
Manticore 7.0.0 92c650401@25013002 (columnar 4.0.0 5aa8e43@25012409) (secondary 4.0.0 5aa8e43@25012409) (knn 4.0.0 5aa8e43@25012409)
Mount success
2025-02-21 10:59:09,695 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2025-02-21 10:59:09,698 INFO RPC interface 'supervisor' initialized
2025-02-21 10:59:09,698 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-02-21 10:59:09,698 INFO supervisord started with pid 42
2025-02-21 10:59:10,700 INFO spawned: 'quorum_recover' with pid 43
2025-02-21 10:59:10,702 INFO spawned: 'searchd_replica' with pid 44
[2025-02-21T10:59:10.741191+00:00] Logs.INFO: Replication mode: multi-master [] []
[2025-02-21T10:59:10.768991+00:00] Logs.INFO: Pods count 0 [] []
[2025-02-21T10:59:10.769050+00:00] Logs.INFO: One pod [] []
2025-02-21 10:59:10,928 INFO spawned: 'searchd' with pid 49
[Fri Feb 21 10:59:10.940 2025] [49] using config file '/etc/manticoresearch/manticore.conf' (672 chars)...
starting daemon version '7.0.0 92c650401@25013002 (columnar 4.0.0 5aa8e43@25012409) (secondary 4.0.0 5aa8e43@25012409) (knn 4.0.0 5aa8e43@25012409)' ...
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for mysql, port=9306
listening on all interfaces for sphinx and http(s), port=9308
listening on all interfaces for VIP mysql, port=9301
listening on 192.168.4.206:9312 for sphinx and http(s)
prereading 0 tables
preread 0 tables in 0.000 sec
accepting connections
[BUDDY] started v3.17.1+25012911-51758b05 'manticore-executor /usr/share/manticore/modules/manticore-buddy/src/main.php --listen=http://0.0.0.0:9308 --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-distributed-insert' at http://127.0.0.1:42475
[BUDDY] Loaded plugins:
[BUDDY] core: empty-string, backup, emulate-elastic, fuzzy, create, drop, insert, alias, select, show, plugin, test, alter-column, alter-distributed-table, alter-rename-table, modify-table, knn, replace, queue, update, autocomplete, cli-table
[BUDDY] local:
[BUDDY] extra:
2025-02-21 10:59:12,136 INFO success: searchd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-21 10:59:12,136 INFO success: quorum_recover entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-21 10:59:12,137 INFO success: searchd_replica entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[2025-02-21T10:59:12.167558+00:00] Logs.INFO: Wait until manti2-manticoresearch-worker-0 came alive [] []
WARNING: No persistent state found. Bootstraping with default state
Crash!!! Handling signal 7
[2025-02-21T11:00:15.480956+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #3 {main} [] []
[2025-02-21T11:00:15.480956+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #3 {main} [] []
[2025-02-21T11:00:15.481068+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
2025-02-21 11:00:15,481 INFO exited: searchd (exit status 2; not expected)
[2025-02-21T11:00:15.481068+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
2025-02-21 11:00:15,483 INFO spawned: 'searchd' with pid 129
[Fri Feb 21 11:00:15.496 2025] [129] using config file '/etc/manticoresearch/manticore.conf' (672 chars)...
starting daemon version '7.0.0 92c650401@25013002 (columnar 4.0.0 5aa8e43@25012409) (secondary 4.0.0 5aa8e43@25012409) (knn 4.0.0 5aa8e43@25012409)' ...
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for mysql, port=9306
listening on all interfaces for sphinx and http(s), port=9308
listening on all interfaces for VIP mysql, port=9301
listening on 192.168.4.206:9312 for sphinx and http(s)
prereading 0 tables
preread 0 tables in 0.000 sec
accepting connections
[BUDDY] started v3.17.1+25012911-51758b05 'manticore-executor /usr/share/manticore/modules/manticore-buddy/src/main.php --listen=http://0.0.0.0:9308 --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-distributed-insert' at http://127.0.0.1:36589
[BUDDY] Loaded plugins:
[BUDDY] core: empty-string, backup, emulate-elastic, fuzzy, create, drop, insert, alias, select, show, plugin, test, alter-column, alter-distributed-table, alter-rename-table, modify-table, knn, replace, queue, update, autocomplete, cli-table
[BUDDY] local:
[BUDDY] extra:
[2025-02-21T11:00:16.481249+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #4 {main} [] []
[2025-02-21T11:00:16.481249+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #4 {main} [] []
[2025-02-21T11:00:16.481334+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:16.481334+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:17.481519+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #5 {main} [] []
[2025-02-21T11:00:17.481519+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #5 {main} [] []
[2025-02-21T11:00:17.481617+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:17.481617+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
2025-02-21 11:00:17,481 INFO success: searchd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[2025-02-21T11:00:18.481799+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #5 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #6 {main} [] []
[2025-02-21T11:00:18.481799+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #5 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #6 {main} [] []
[2025-02-21T11:00:18.481901+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:18.481901+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:19.482084+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #5 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #6 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #7 {main} [] []
[2025-02-21T11:00:19.482084+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #5 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #6 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #7 {main} [] []
[2025-02-21T11:00:19.482182+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:19.482182+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []