ManticoreSearch / replication on Kubernetes

Dear all,

I am successfully running a Manticore instance on Kubernetes using Manticore’s latest Docker Image (3.5.4). Unfortunately, I’m facing an issue trying to set up a replication Cluster.

I have created two pods (running on two differents nodes) named manticore1 and manticore2, with the following config:

searchd
{
listen = 9306:mysql41
listen = 9312
listen = 9322:replication
log = /var/log/manticore/query.log
query_log = /var/log/manticore/searchd.log
pid_file = /var/run/manticore/searchd.pid
preopen_indexes = 1
binlog_path = /var/lib/manticore
data_dir = /var/lib/manticore
collation_server = utf8_general_ci
max_packet_size = 64M
max_open_files = max
}

The ports 9306, 9312 and 9322 are opened on each pod’s service.

I am able to connect to each of them using their “kubernetes” name :
mysql -h manticore1
mysql -h manticore2

On the first node, I am doing this :

  • CREATE CLUSTER xxx;
  • ALTER CLUSTER xxx ADD index1;
  • ALTER CLUSTER xxx ADD index2;

When I issue the following query on Manticore2, it hangs and in the end I have this error:
JOIN CLUSTER profiles AT ‘manticore1:9312’;
ERROR 1064 (42000): replication connection failed: 7 ‘error in node state, must reinit’

Unfortunately, I can’t issue an IP address in the config file, is it mandatory?
Did I forget something?

Best regards,
JF

PS : Here is the output of SHOW STATUS on the first node :

mysql> show status;
+---------------------------------------------+-------------------------------------------+
| Counter                                     | Value                                     |
+---------------------------------------------+-------------------------------------------+
| uptime                                      | 745                                       |
| connections                                 | 1716                                      |
| maxed_out                                   | 0                                         |
| version                                     | 3.5.4 13f8d08d@201211 release             |
| mysql_version                               | 3.5.4 13f8d08d@201211 release             |
| command_search                              | 0                                         |
| command_excerpt                             | 0                                         |
| command_update                              | 0                                         |
| command_keywords                            | 0                                         |
| command_persist                             | 0                                         |
| command_status                              | 2                                         |
| command_flushattrs                          | 0                                         |
| command_sphinxql                            | 0                                         |
| command_ping                                | 0                                         |
| command_delete                              | 0                                         |
| command_set                                 | 0                                         |
| command_insert                              | 0                                         |
| command_replace                             | 0                                         |
| command_commit                              | 1712                                      |
| command_suggest                             | 0                                         |
| command_json                                | 0                                         |
| command_callpq                              | 0                                         |
| command_clusterpq                           | 0                                         |
| command_getfield                            | 1                                         |
| agent_connect                               | 0                                         |
| agent_tfo                                   | 0                                         |
| agent_retry                                 | 0                                         |
| queries                                     | 0                                         |
| dist_queries                                | 0                                         |
| workers_total                               | 2                                         |
| workers_active                              | 3430                                      |
| workers_clients                             | 1714                                      |
| work_queue_length                           | 1716                                      |
| query_wall                                  | 0.000                                     |
| query_cpu                                   | OFF                                       |
| dist_wall                                   | 0.000                                     |
| dist_local                                  | 0.000                                     |
| dist_wait                                   | 0.000                                     |
| query_reads                                 | OFF                                       |
| query_readkb                                | OFF                                       |
| query_readtime                              | OFF                                       |
| avg_query_wall                              | 0.000                                     |
| avg_query_cpu                               | OFF                                       |
| avg_dist_wall                               | 0.000                                     |
| avg_dist_local                              | 0.000                                     |
| avg_dist_wait                               | 0.000                                     |
| avg_query_reads                             | OFF                                       |
| avg_query_readkb                            | OFF                                       |
| avg_query_readtime                          | OFF                                       |
| qcache_max_bytes                            | 16777216                                  |
| qcache_thresh_msec                          | 3000                                      |
| qcache_ttl_sec                              | 60                                        |
| qcache_cached_queries                       | 0                                         |
| qcache_used_bytes                           | 0                                         |
| qcache_hits                                 | 0                                         |
| cluster_name                                | profiles                                  |
| cluster_profiles_state_uuid                 | b929325a-451b-11eb-9d65-47cbb4afc64e      |
| cluster_profiles_conf_id                    | 1                                         |
| cluster_profiles_status                     | primary                                   |
| cluster_profiles_size                       | 1                                         |
| cluster_profiles_local_index                | 0                                         |
| cluster_profiles_node_state                 | synced                                    |
| cluster_profiles_nodes_set                  |                                           |
| cluster_profiles_nodes_view                 | 127.0.0.1:9312,127.0.0.1:9322:replication |
| cluster_profiles_indexes_count              | 2                                         |
| cluster_profiles_indexes                    | digsty,suggests                           |
| cluster_profiles_local_state_uuid           | b929325a-451b-11eb-9d65-47cbb4afc64e      |
| cluster_profiles_protocol_version           | 9                                         |
| cluster_profiles_last_applied               | 2                                         |
| cluster_profiles_last_committed             | 2                                         |
| cluster_profiles_replicated                 | 2                                         |
| cluster_profiles_replicated_bytes           | 288                                       |
| cluster_profiles_repl_keys                  | 2                                         |
| cluster_profiles_repl_keys_bytes            | 64                                        |
| cluster_profiles_repl_data_bytes            | 86                                        |
| cluster_profiles_repl_other_bytes           | 0                                         |
| cluster_profiles_received                   | 2                                         |
| cluster_profiles_received_bytes             | 178                                       |
| cluster_profiles_local_commits              | 0                                         |
| cluster_profiles_local_cert_failures        | 0                                         |
| cluster_profiles_local_replays              | 0                                         |
| cluster_profiles_local_send_queue           | 0                                         |
| cluster_profiles_local_send_queue_max       | 1                                         |
| cluster_profiles_local_send_queue_min       | 0                                         |
| cluster_profiles_local_send_queue_avg       | 0.000000                                  |
| cluster_profiles_local_recv_queue           | 0                                         |
| cluster_profiles_local_recv_queue_max       | 2                                         |
| cluster_profiles_local_recv_queue_min       | 0                                         |
| cluster_profiles_local_recv_queue_avg       | 0.500000                                  |
| cluster_profiles_local_cached_downto        | 1                                         |
| cluster_profiles_flow_control_paused_ns     | 0                                         |
| cluster_profiles_flow_control_paused        | 0.000000                                  |
| cluster_profiles_flow_control_sent          | 0                                         |
| cluster_profiles_flow_control_recv          | 0                                         |
| cluster_profiles_flow_control_interval      | [ 100, 100 ]                              |
| cluster_profiles_flow_control_interval_low  | 100                                       |
| cluster_profiles_flow_control_interval_high | 100                                       |
| cluster_profiles_flow_control_status        | OFF                                       |
| cluster_profiles_cert_deps_distance         | 1.000000                                  |
| cluster_profiles_apply_oooe                 | 0.000000                                  |
| cluster_profiles_apply_oool                 | 0.000000                                  |
| cluster_profiles_apply_window               | 1.000000                                  |
| cluster_profiles_commit_oooe                | 0.000000                                  |
| cluster_profiles_commit_oool                | 0.000000                                  |
| cluster_profiles_commit_window              | 1.000000                                  |
| cluster_profiles_local_state                | 4                                         |
| cluster_profiles_local_state_comment        | Synced                                    |
| cluster_profiles_cert_index_size            | 2                                         |
| cluster_profiles_cert_bucket_count          | 4                                         |
| cluster_profiles_gcache_pool_size           | 1688                                      |
| cluster_profiles_causal_reads               | 0                                         |
| cluster_profiles_cert_interval              | 0.000000                                  |
| cluster_profiles_open_transactions          | 0                                         |
| cluster_profiles_open_connections           | 0                                         |
| cluster_profiles_ist_receive_status         |                                           |
| cluster_profiles_ist_receive_seqno_start    | 0                                         |
| cluster_profiles_ist_receive_seqno_current  | 0                                         |
| cluster_profiles_ist_receive_seqno_end      | 0                                         |
| cluster_profiles_incoming_addresses         | 127.0.0.1:9312,127.0.0.1:9322:replication |
| cluster_profiles_cluster_weight             | 1                                         |
| cluster_profiles_desync_count               | 0                                         |
| cluster_profiles_evs_delayed                |                                           |
| cluster_profiles_evs_evict_list             |                                           |
| cluster_profiles_evs_repl_latency           | 0/0/0/0/0                                 |
| cluster_profiles_evs_state                  | OPERATIONAL                               |
| cluster_profiles_gcomm_uuid                 | b928f5ce-451b-11eb-af8b-a2db647dffd6      |

I suspect it’s because the sphinxapi and replication ports are listening on lo interface instead of the external.
Try something in config like

#!/bin/sh
ip=`hostname -i`
cat << EOF
searchd {
....
    listen = $ip:9312
    listen = $ip:9322-9325:replication
....
}
EOF

Thanks, I finally made it with a similar hack :slight_smile:

#!/usr/bin/php

searchd
{
        listen                  = 9306:mysql41
        listen                  = <?= gethostname() ?>:9312
        listen                  = <?= gethostname() ?>:9315-9325:replication
        log                     = /var/log/manticore/query.log
        query_log               = /var/log/manticore/searchd.log
        pid_file                = /var/run/manticore/searchd.pid
        preopen_indexes         = 1
        binlog_path             = /var/lib/manticore
        data_dir                = /var/lib/manticore
        collation_server        = utf8_general_ci
        max_packet_size = 128M
        max_open_files = max
} 

PS : Link to the forums from the documentation section is dead (because different subdomain)!

1 Like