centos7 docker start manticore, mysql command not connecting

tong_yaoyu · May 13, 2025, 2:06am

I tried to start manticore on centos 7.8 and ubuntu 22.04 using the same docker-compose.yml and manticore.conf with the following configuration.I’m using windows on an intel chip, linux installed in vmware.
docker-compose.yml

version: '2.2'

services:
  manticore:
    container_name: manticore
    image: manticore:1.0
    restart: always
    ports:
      - 9306:9306
      - 9308:9308
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /home/tyy/manticore/manticore-data:/var/lib/manticore
      - /home/tyy/manticore/manticore.conf:/etc/manticoresearch/manticore.conf

manticore.conf

# Other tables omitted
source mysql_tb_platform_api_chain_log
{
    type               = mysql
    sql_host           = 10.100.3.235
    sql_port           = 3306
    sql_user           = root
    sql_pass           = cycad@123
    sql_db             = cycad_logs_202505
    sql_query          = \
      SELECT \
        log_id AS id, \
        log_param, \
        cast(UNCOMPRESS(log_text) AS char) AS log_text, \
        UNIX_TIMESTAMP(createdAt) AS created_at, \
        UNIX_TIMESTAMP(updatedAt) AS updated_at \
      FROM tb_platform_api_chain_log \
      WHERE created_at between "2025-05-01 00:00:00" and "2025-05-01 00:59:59"
    sql_field_string   = log_param
    sql_field_string   = log_text
    sql_attr_timestamp = created_at
    sql_attr_timestamp = updated_at
}

table plain_tb_platform_api_chain_log
{
    type             = plain
    source           = mysql_tb_platform_api_chain_log
    path             = /var/lib/manticore/plain_tb_platform_api_chain_log
    charset_table    = non_cjk,chinese
    stopwords        = en,zh
    morphology       = icu_chinese,stem_en
}

table rt_tb_platform_api_chain_log {
    type = rt
    path = /var/lib/manticore/rt_tb_platform_api_chain_log
    rt_attr_bigint    = id
    rt_field          = log_param
    rt_field          = log_text
    rt_attr_timestamp = created_at
    rt_attr_timestamp = updated_at
    charset_table     = non_cjk,chinese
    morphology        = icu_chinese,stem_en
    stopwords         = en,zh
}

searchd {
    listen = 0.0.0.0:9312
    listen = 0.0.0.0:9306:mysql
    listen = 0.0.0.0:9308:http
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
}

Both systems have the same startup configuration, but centos can’t connect using the mysql client, while ubuntu succeeds directly, the screenshot is as follows
centos7.8

ubuntu22.04

Centos did the same thing when I tried it a year and a half ago, and I’ve posted about it here, and now centos is still unsuccessful in trying the new version, and I don’t know what’s causing it, and I don’t know if it’s centos or not… any gurus know what’s causing it? What should I do to fix it? My m4 chip Mac, the same configuration can also run normally，Or maybe I should just give up on centos.

Sergey · May 13, 2025, 5:14am

It’s not clear what is:

image: manticore:1.0

Can you provide the Dockerfiles?

I’d also check in /var/log/manticore/searchd.log to ensure Manticore starts fine at all in Centos. We test Manticore nightly in Centos 7 and it works fine - 🌙 Nightly tests 83c622b66f06acc35f51af0621cc380390b3cffb · manticoresoftware/manticoresearch@83c622b · GitHub

tong_yaoyu · May 13, 2025, 6:01am

I actually downloaded the latest manticore_amd64 version of the image in my mac, and then export as tar package, and then imported it into linux docker, and then docker tag command packaged the name, the actual version is the latest, the following is the screenshot of ubuntu

in centos check searchd.log

It looks like nothing is wrong, but the mysql command with manticore gives an error ERROR 2003 (HY000): Can’t connect to MySQL server on ‘127.0.0.1:9306’ (111)

Sergey · May 13, 2025, 6:32am

downloaded the latest manticore_amd64 version of the image in my mac, and then export as tar package, and then imported it into linux docker, and then docker tag command packaged the name

the latest manticore_amd64 docker image is FROM ubuntu:jammy. At what point does it turn into Centos? I don’t get it.

tong_yaoyu · May 13, 2025, 6:42am

I just executed the command in centos

docker pull manticoresearch/manticore

The downloaded image is the same as the one I imported，the image id is the same

I used this image to restart the manticore container, docker-compose.yml is as follows

version: '2.2'

services:
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    restart: always
    ports:
      - 9306:9306
      - 9308:9308
      - 9312:9312
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /tyy/manticore/manticore.conf:/etc/manticoresearch/manticore.conf
      - /tyy/manticore/manticore-data:/var/lib/manticore

Then enter the container and try to connect to manticore, but it still doesn’t work

searchd.log as follows

Is it an image issue please? does docker pull manticoresearch/manticore download the ubuntu image by default?

Sergey · May 13, 2025, 6:56am

That’s for sure since there’s no centos-based official Manticore docker image.

I just executed the command in centos
docker pull manticoresearch/manticore

but previously you said:

downloaded the latest manticore_amd64 version of the image in my mac, and then export as tar package, and then imported it into linux docker, and then docker tag command packaged the name

I’m a bit confused about what exactly you did. Can you confirm the following?

You have a physical server running CentOS 7
You run docker pull manticoresearch/manticore on that server
Then you access the container using docker exec
After that, running mysql -P9306 -h 127.0.0.1 inside the container doesn’t work

?

tong_yaoyu · May 13, 2025, 7:03am

Yes! You’re absolutely right, those four points are what I’m doing right now. I re-downloaded the official image in centos and re-run the new container, but I still can’t connect to manticore, with the same error I had in the screenshot before.

Does this mean that if I want to run manticore in linux with docker, I can only use ubuntu? At least I’m having the above problem with centos right now to run the image of mantciore.

Sergey · May 13, 2025, 7:16am

What does telnet 127.0.0.1 9306 return in the container on the Centos server?

tong_yaoyu · May 13, 2025, 7:29am

here is the result

and this is the results of other commands

the results are a little strange

Sergey · May 13, 2025, 8:00am

It means that searchd doesn’t listen on 127.0.0.1:9306. Try to run something else instead of Manticore which would listen on 0.0.0.0:9306 (e.g. nc) to check if it has smth to do with Manticore at all.

tong_yaoyu · May 13, 2025, 8:15am

I ran nc -l 9306 inside the container

Then telnet 127.0.0.1 9306 was executed on the centos host.

It’s possible to communicate.
does this mean it’s a problem with manticore?

Sergey · May 13, 2025, 8:17am

What does netstat -tuln return when running nc ?

tong_yaoyu · May 13, 2025, 8:19am

here is the result

Sergey · May 13, 2025, 8:52am

So nc does listen on 0.0.0.0:9306 while Manticore doesn’t. That’s strange. Can you reproduce this issue in Centos 8? Centos 7 is too old and it seems not possible to run it in Hetzner Cloud. When I try centos:7 + manticore inside it - it works fine to me:

[root@ea11f0f39649 /]# cat /etc/*release*|grep CentOS
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
PRETTY_NAME="CentOS Linux 7 (Core)"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)
[root@ea11f0f39649 /]# netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:42237         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9308            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9306            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9312            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.11:33057        0.0.0.0:*               LISTEN
udp        0      0 127.0.0.11:55532        0.0.0.0:*
[root@ea11f0f39649 /]# mysql -P9306 -h127.0.0.1
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 9.3.2 428075261@25050208 (columnar 4.2.1 0fe99f4@25050206) (secondary 4.2.1 0fe99f4@25050206) (knn 4.2.1 0fe99f4@25050206) git branch manticore-9.3.2...origin/manticore-9.3.2

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>

tong_yaoyu · May 13, 2025, 9:00am

Sergey:

[root@ea11f0f39649 /]# cat /etc/*release*|grep CentOS
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
PRETTY_NAME="CentOS Linux 7 (Core)"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)
[root@ea11f0f39649 /]# netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:42237         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9308            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9306            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9312            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.11:33057        0.0.0.0:*               LISTEN
udp        0      0 127.0.0.11:55532        0.0.0.0:*
[root@ea11f0f39649 /]# mysql -P9306 -h127.0.0.1
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 9.3.2 428075261@25050208 (columnar 4.2.1 0fe99f4@25050206) (secondary 4.2.1 0fe99f4@25050206) (knn 4.2.1 0fe99f4@25050206) git branch manticore-9.3.2...origin/manticore-9.3.2

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>

Is this centos 7.9 manticore running with docker or is it a yum install running manticore? docker running manticore may be different.

I need to download a centos8, give me a moment, I’ll try it

Sergey · May 13, 2025, 10:15am

This is CentOS 7.9 running in Docker with Manticore installed via yum install manticore. In other words, it’s not using the official Manticore Docker image.

tong_yaoyu · May 13, 2025, 1:28pm

I tried centos8, with the same startup configuration, msyql command still can’t connect manticore, the same error as centos7

[root@localhost manticore]# cat /etc/*release*|grep CentOS
CentOS Linux release 8.5.2111
NAME="CentOS Linux"
PRETTY_NAME="CentOS Linux 8"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CentOS Linux release 8.5.2111
CentOS Linux release 8.5.2111
[root@localhost manticore]# docker exec -it manticore /bin/bash
root@d0dda4019a03:/var/lib/manticore# mysql -P9306 -h 127.0.0.1
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:9306' (111)
root@d0dda4019a03:/var/lib/manticore# mysql -P9306 -h0
ERROR 2003 (HY000): Can't connect to MySQL server on '0:9306' (111)
root@d0dda4019a03:/var/lib/manticore# telnet 127.0.0.1 9306
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Again using nc -l 9306 inside the container gives the following result，same result as above

root@d0dda4019a03:/var/lib/manticore# netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.11:34121        0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:9306            0.0.0.0:*               LISTEN     
udp        0      0 127.0.0.11:36762        0.0.0.0:*

You can try to start manticore in centos with a docker image of manticore, I think the result should be the same as mine.

Is there some solution? If there’s no solution, it looks like I’ll have to give up on centos…

Sergey · May 13, 2025, 4:58pm

I couldn’t find a VPS with CentOS 8, but I tested it on an AlmaLinux 8 VPS, which should be very similar to CentOS 8.

[root@alma-4gb-hel1-2 ~]# cat /etc/*release*
AlmaLinux release 8.10 (Cerulean Leopard)
...

[root@alma-4gb-hel1-2 ~]# docker run -d --name=manticore --rm manticoresearch/manticore:latest
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
91d67b7f5f61fdb6ff18b7b1008526005b88c9f047d9fd462df866fae112aa80

[root@alma-4gb-hel1-2 ~]# docker exec -it manticore bash
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
root@91d67b7f5f61:/var/lib/manticore# mysql -P9306 -h0
mysql>

Can you try the exact same thing on your CentOS 8 setup (without your config or docker-compose) and confirm if it still doesn’t work?

tong_yaoyu · May 14, 2025, 1:43am

Here’s what I tried in centos8, and it still didn’t work as before

[root@localhost manticore]# docker run -d --name=manticore --rm manticoresearch/manticore:latest
56e93dcebc18dd203a52c72efe75ca3d47ade817bff05fcdbdc90526aa794035
[root@localhost manticore]# docker exec -it manticore bash
root@56e93dcebc18:/var/lib/manticore# mysql -P9306 -h0
ERROR 2003 (HY000): Can't connect to MySQL server on '0:9306' (111)
root@56e93dcebc18:/var/lib/manticore# mysql -P9306 -h127.0.0.1
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:9306' (111)

I see you have centos 7.9 here, It’s running in docker though, not sure if it can install another docker in this, if so，you can try starting manticore with a docker image here to see if you can connect successfully
This website is official iso. If you have time, you can also try centos7 or 8

I’ll try AlmaLinux8 and see if this works

Sergey · May 14, 2025, 8:52am

Here’s what I tried in centos8, and it still didn’t work as before

This is really strange. I can’t believe there’s some kind of issue between CentOS 8 and AlmaLinux 8.

I see you have centos 7.9 here, It’s running in docker though, not sure if it can install another docker in this

Right. That’s the problem. There’s this dind image https://hub.docker.com/_/docker , but it’s based not on Centos and I’m not sure how difficult it would be to remake it to be based on Centos 7. Also, docker-in-docker may be different from docker on a server in terms of networking.

I’ll try AlmaLinux8 and see if this works

Thanks! I’ll wait to hear back from you.