Help on deploying manticore on K8s cluster

Cristian_Lezau · February 20, 2025, 5:43pm

I am trying to build a POC with a manticore multimaster instance & persistent volumes stored on a NFS instance.
For storage i am using nutanix csi.
I have downloaded the Helm chart in my Rancher ui and i am trying to deploy with 3 replica’s.
There is no replica deployed because of:
[2025-02-20T17:12:41.338130+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away.
I can get this running on a “normal” where i can edit manticore.conf , but in the k8s scenario i don’t see how i can do it.
please help, i’m stuck.

Sergey · February 21, 2025, 2:23am

Hi @Cristian_Lezau

Please provide more logs. The error MySQL server has gone away could be due to a timeout or a crash. It’s hard to determine the cause without more details.

Cristian_Lezau · February 21, 2025, 11:06am

Thanks @Sergey .Here is the full log. I hope that this is the one you are looking for. If not let me know what else is needed and i will happily provide:

Manticore 7.0.0 92c650401@25013002 (columnar 4.0.0 5aa8e43@25012409) (secondary 4.0.0 5aa8e43@25012409) (knn 4.0.0 5aa8e43@25012409)
Mount success
2025-02-21 10:59:09,695 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2025-02-21 10:59:09,698 INFO RPC interface 'supervisor' initialized
2025-02-21 10:59:09,698 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-02-21 10:59:09,698 INFO supervisord started with pid 42
2025-02-21 10:59:10,700 INFO spawned: 'quorum_recover' with pid 43
2025-02-21 10:59:10,702 INFO spawned: 'searchd_replica' with pid 44
[2025-02-21T10:59:10.741191+00:00] Logs.INFO: Replication mode: multi-master [] []
[2025-02-21T10:59:10.768991+00:00] Logs.INFO: Pods count 0 [] []
[2025-02-21T10:59:10.769050+00:00] Logs.INFO: One pod [] []
2025-02-21 10:59:10,928 INFO spawned: 'searchd' with pid 49
[Fri Feb 21 10:59:10.940 2025] [49] using config file '/etc/manticoresearch/manticore.conf' (672 chars)...
starting daemon version '7.0.0 92c650401@25013002 (columnar 4.0.0 5aa8e43@25012409) (secondary 4.0.0 5aa8e43@25012409) (knn 4.0.0 5aa8e43@25012409)' ...
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for mysql, port=9306
listening on all interfaces for sphinx and http(s), port=9308
listening on all interfaces for VIP mysql, port=9301
listening on 192.168.4.206:9312 for sphinx and http(s)
prereading 0 tables
preread 0 tables in 0.000 sec
accepting connections
[BUDDY] started v3.17.1+25012911-51758b05 'manticore-executor /usr/share/manticore/modules/manticore-buddy/src/main.php --listen=http://0.0.0.0:9308   --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-distributed-insert' at http://127.0.0.1:42475
[BUDDY] Loaded plugins:
[BUDDY]   core: empty-string, backup, emulate-elastic, fuzzy, create, drop, insert, alias, select, show, plugin, test, alter-column, alter-distributed-table, alter-rename-table, modify-table, knn, replace, queue, update, autocomplete, cli-table
[BUDDY]   local: 
[BUDDY]   extra: 
2025-02-21 10:59:12,136 INFO success: searchd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-21 10:59:12,136 INFO success: quorum_recover entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-21 10:59:12,137 INFO success: searchd_replica entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[2025-02-21T10:59:12.167558+00:00] Logs.INFO: Wait until manti2-manticoresearch-worker-0 came alive [] []
WARNING: No persistent state found. Bootstraping with default state
Crash!!! Handling signal 7
[2025-02-21T11:00:15.480956+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #3 {main} [] []
[2025-02-21T11:00:15.480956+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #3 {main} [] []
[2025-02-21T11:00:15.481068+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
2025-02-21 11:00:15,481 INFO exited: searchd (exit status 2; not expected)
[2025-02-21T11:00:15.481068+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
2025-02-21 11:00:15,483 INFO spawned: 'searchd' with pid 129
[Fri Feb 21 11:00:15.496 2025] [129] using config file '/etc/manticoresearch/manticore.conf' (672 chars)...
starting daemon version '7.0.0 92c650401@25013002 (columnar 4.0.0 5aa8e43@25012409) (secondary 4.0.0 5aa8e43@25012409) (knn 4.0.0 5aa8e43@25012409)' ...
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for mysql, port=9306
listening on all interfaces for sphinx and http(s), port=9308
listening on all interfaces for VIP mysql, port=9301
listening on 192.168.4.206:9312 for sphinx and http(s)
prereading 0 tables
preread 0 tables in 0.000 sec
accepting connections
[BUDDY] started v3.17.1+25012911-51758b05 'manticore-executor /usr/share/manticore/modules/manticore-buddy/src/main.php --listen=http://0.0.0.0:9308   --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-distributed-insert' at http://127.0.0.1:36589
[BUDDY] Loaded plugins:
[BUDDY]   core: empty-string, backup, emulate-elastic, fuzzy, create, drop, insert, alias, select, show, plugin, test, alter-column, alter-distributed-table, alter-rename-table, modify-table, knn, replace, queue, update, autocomplete, cli-table
[BUDDY]   local: 
[BUDDY]   extra: 
[2025-02-21T11:00:16.481249+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #4 {main} [] []
[2025-02-21T11:00:16.481249+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #4 {main} [] []
[2025-02-21T11:00:16.481334+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:16.481334+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:17.481519+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #5 {main} [] []
[2025-02-21T11:00:17.481519+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #5 {main} [] []
[2025-02-21T11:00:17.481617+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:17.481617+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
2025-02-21 11:00:17,481 INFO success: searchd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[2025-02-21T11:00:18.481799+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #5 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #6 {main} [] []
[2025-02-21T11:00:18.481799+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #5 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #6 {main} [] []
[2025-02-21T11:00:18.481901+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:18.481901+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:19.482084+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #5 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #6 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #7 {main} [] []
[2025-02-21T11:00:19.482084+00:00] Logs.ERROR: Exception until query processing. Query: CREATE CLUSTER manticore_cluster . Error: mysqli_sql_exception: MySQL server has gone away in /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php:33 Stack trace: #0 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(33): mysqli->query() #1 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #2 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #3 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #4 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreMysqliFetcher.php(51): Core\Manticore\ManticoreMysqliFetcher->query() #5 /etc/manticoresearch/vendor/manticoresoftware/manticoresearch-auto-replication/src/Manticore/ManticoreConnector.php(120): Core\Manticore\ManticoreMysqliFetcher->query() #6 /etc/manticoresearch/replica.php(116): Core\Manticore\ManticoreConnector->createCluster() #7 {main} [] []
[2025-02-21T11:00:19.482182+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []
[2025-02-21T11:00:19.482182+00:00] Logs.ERROR: Error until query processing. Query: CREATE CLUSTER manticore_cluster . Error: MySQL server has gone away [] []

Sergey · February 23, 2025, 2:22am

Thanks for the details @Cristian_Lezau
However it’s still not clear what’s going on.

Sergey · February 23, 2025, 2:23am

Crash!!! Handling signal 7

Hm, perhaps a common reason with:

github.com/kubernetes/ingress-nginx

"worker process <process-id> exited on signal 7 (core dumped)" using geoip2 module

opened 09:51AM - 21 Apr 22 UTC

closed 01:29PM - 21 Apr 22 UTC

eneamarcantoni99

needs-kind needs-triage needs-priority

**NGINX Ingress controller version** (exec into the pod and run nginx-ingress-co…ntroller --version.): Release: v1.1.3 Build: 9d3a285f19a704524439c75b947e2189406565ab Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.19.10 **Kubernetes version** (use `kubectl version`): Major: 1 Minor: 21+ GitVersion: v1.21.5-eks-bc4871b GitCommit: 5236faf39f1b7a7dabea8df12726f25608131aa9 GitTreeState: clean BuildDate: 2021-10-29T23:32:16Z GoVersion: go1.16.8 Compiler: gc Platform: linux/amd64 **Environment**: - **Cloud provider or hardware configuration**: AWS EKS - **OS** (e.g. from /etc/os-release): Amazon Linux 2 - **Kernel** (e.g. `uname -a`): 5.4.181-99.354.amzn2.x86_64 - **Install tools**: - We have deployed Kubernetes Cluster using EKS Console. - **How was the ingress-nginx-controller installed**: - Ingress controller was installed running the following command: kubectl apply -f [baremetal-deploy.yaml.txt](https://github.com/kubernetes/ingress-nginx/files/8529689/baremetal-deploy.yaml.txt) **What happened**: randomly the ingress controller become unresponsive and start logging a lot of error like "worker process 991 exited on signal 7 (core dumped)". Reading a core dump using GDB I can see that the BUS error is caused from libmaxminddb.so.0. I've attached the output of gdb: [gdb.txt](https://github.com/kubernetes/ingress-nginx/files/8529830/gdb.txt)

github.com/leev/ngx_http_geoip2_module

Nginx returns 502 'worker process exited on signal 7'

opened 10:11PM - 18 Jun 21 UTC

ysgard

This is an odd one and I can't reproduce it reliably, but it happens often enoug…h in our test/staging environments (but not production) that we've had to stop using the module. Signal 7 is SIGBUS and usually a module problem when it occurs in nginx. What happens is that occasionally nginx will wedge on a request, returning a 502. The log contains this cryptic error: `worker process 8519 exited on signal 7` The only reason I know it's GeoIP2 is because when the nginx is recompiled without the geoip2 module, it goes away. I've tried both static and dynamic compilations and the result is the same. I've put the nginx configure script and nginx.conf in a gist: https://gist.github.com/ysgard/bd8305af0689af5484c21230a2cbf0fc The problem is odd because it only manifests on our environments which don't get a lot of load/traffic. On our production systems it runs without issue.

Sergey · February 23, 2025, 2:25am

How do you reproduce the issue? In what environment does it happen? Can you reproduce it if you run some k3s or kind or minikube?

Cristian_Lezau · February 24, 2025, 9:30am

i am using rancher to manage the cluster and i am deploying the helm chart with default configuration (exception is storage class)
here is the printed worker config:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/config: 2512344ff793e810b8df9dece4fc6783792343354d69122160139068929b5643
    cni.projectcalico.org/containerID: 4e0dca80c13851ff97995d99ce3976bf87e098159e8d8c9c39861697d8dfd6fa
    cni.projectcalico.org/podIP: 192.168.34.23/32
    cni.projectcalico.org/podIPs: 192.168.34.23/32
  creationTimestamp: "2025-02-24T09:08:37Z"
  generateName: manti2-manticoresearch-worker-
  labels:
    app.kubernetes.io/component: worker
    app.kubernetes.io/instance: manti2
    app.kubernetes.io/name: manticoresearch
    apps.kubernetes.io/pod-index: "0"
    controller-revision-hash: manti2-manticoresearch-worker-f7b6c4b9f
    name: manti2-manticoresearch-worker
    statefulset.kubernetes.io/pod-name: manti2-manticoresearch-worker-0
  name: manti2-manticoresearch-worker-0
  namespace: manticore
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: manti2-manticoresearch-worker
    uid: 1f34b224-5d8a-41f2-abc0-63b527e05dbd
  resourceVersion: "1124964"
  uid: 14968ee1-4518-408b-9133-87198054489d
spec:
  containers:
  - env:
    - name: POD_START_VIA_PROBE
      value: "true"
    - name: AUTO_ADD_TABLES_IN_CLUSTER
      value: "true"
    - name: IS_BALANCER_ENABLED
      value: "true"
    - name: CONFIGMAP_PATH
      value: /mnt/manticore.conf
    - name: MANTICORE_PORT
      value: "9306"
    - name: MANTICORE_BINARY_PORT
      value: "9312"
    - name: CLUSTER_NAME
      value: manticore
    - name: REPLICATION_MODE
      value: multi-master
    - name: LOG_LEVEL
      value: INFO
    - name: INSTANCE_LABEL
      value: manti2
    - name: WORKER_SERVICE
      value: manti2-manticoresearch-worker-replication-svc
    - name: NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: EXTRA
      value: "1"
    image: manticoresearch/helm-worker:7.0.0.1
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - ./shutdown.sh
"/tmp/kubectl-edit-1003015580.yaml" 203L, 6081B                                                                                                                                                                               1,1           Top
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/config: 2512344ff793e810b8df9dece4fc6783792343354d69122160139068929b5643
    cni.projectcalico.org/containerID: 4e0dca80c13851ff97995d99ce3976bf87e098159e8d8c9c39861697d8dfd6fa
    cni.projectcalico.org/podIP: 192.168.34.23/32
    cni.projectcalico.org/podIPs: 192.168.34.23/32
  creationTimestamp: "2025-02-24T09:08:37Z"
  generateName: manti2-manticoresearch-worker-
  labels:
    app.kubernetes.io/component: worker
    app.kubernetes.io/instance: manti2
    app.kubernetes.io/name: manticoresearch
    apps.kubernetes.io/pod-index: "0"
    controller-revision-hash: manti2-manticoresearch-worker-f7b6c4b9f
    name: manti2-manticoresearch-worker
    statefulset.kubernetes.io/pod-name: manti2-manticoresearch-worker-0
  name: manti2-manticoresearch-worker-0
  namespace: manticore
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: manti2-manticoresearch-worker
    uid: 1f34b224-5d8a-41f2-abc0-63b527e05dbd
  resourceVersion: "1124964"
  uid: 14968ee1-4518-408b-9133-87198054489d
spec:
  containers:
  - env:
    - name: POD_START_VIA_PROBE
      value: "true"
    - name: AUTO_ADD_TABLES_IN_CLUSTER
      value: "true"
    - name: IS_BALANCER_ENABLED
      value: "true"
    - name: CONFIGMAP_PATH
      value: /mnt/manticore.conf
    - name: MANTICORE_PORT
      value: "9306"
    - name: MANTICORE_BINARY_PORT
      value: "9312"
    - name: CLUSTER_NAME
      value: manticore
    - name: REPLICATION_MODE
      value: multi-master
    - name: LOG_LEVEL
      value: INFO
    - name: INSTANCE_LABEL
      value: manti2
    - name: WORKER_SERVICE
      value: manti2-manticoresearch-worker-replication-svc
    - name: NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: EXTRA
      value: "1"
    image: manticoresearch/helm-worker:7.0.0.1
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - ./shutdown.sh
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 5
      periodSeconds: 3
      successThreshold: 1
      tcpSocket:
        port: 9306
      timeoutSeconds: 1
    name: worker
    resources: {}
    securityContext: {}
    startupProbe:
      exec:
        command:
        - /bin/sh
        - -c
        - |
          if /usr/bin/mysql -e "show status;" | grep cluster_node_state | grep -q synced ; then exit 0; else exit 1; fi
      failureThreshold: 30
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/manticore/
      name: data
    - mountPath: /mnt/manticore.conf
      name: config-volume
      subPath: manticore.conf
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-ppjx9
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: manti2-manticoresearch-worker-0
  nodeName: dxkube2
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: manticore-sa
  serviceAccountName: manticore-sa
  subdomain: manti2-manticoresearch-worker-replication-svc
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-manti2-manticoresearch-worker-0
  - configMap:
      defaultMode: 420
      name: manti2-manticoresearch-worker-config
    name: config-volume
  - name: kube-api-access-ppjx9
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-02-24T09:08:37Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-02-24T09:08:37Z"
    message: 'containers with unready status: [worker]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-02-24T09:08:37Z"
    message: 'containers with unready status: [worker]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-02-24T09:08:37Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://4981f511120f4ed78a09b937c8bfe1fdc22fe7f24aedf583d6c2be11c815680a
    image: docker.io/manticoresearch/helm-worker:7.0.0.1
    imageID: docker.io/manticoresearch/helm-worker@sha256:99f894ddbf311e01b39a0faf9562ba04381c629d89f10e02fb8676512ec9631c
    lastState:
      terminated:
        containerID: containerd://ed8702d5d8ac18756a68d4bbaccff0e9ab4c1e001f4935964d45723f4e6d0c0d
        exitCode: 137
        finishedAt: "2025-02-24T09:19:39Z"
        reason: Error
        startedAt: "2025-02-24T09:14:09Z"
    name: worker
    ready: false
    restartCount: 2
    started: false
    state:
      running:
        startedAt: "2025-02-24T09:19:39Z"
  hostIP: 10.113.24.20
  phase: Running
  podIP: 192.168.34.23
  podIPs:
  - ip: 192.168.34.23
  qosClass: BestEffort
  startTime: "2025-02-24T09:08:37Z

Shouldn’t i have also the mysql port printed?