Docker Manticore setup

General Discussions - English
1

I am trying to setup Manticore search within a docker container and I am running into an issues. I am able to build manticore, web node and a mysql container with no problem. Once the container is built it searches for the MySQL database. After that has been completed I am running into this issue

2023-11-28 16:04:38 using config file '/etc/manticoresearch/manticore.conf'...
2023-11-28 16:04:38 indexing table 'cw_issue_pages'...
2023-11-28 16:04:38 collected 0 docs, 0.0 MB
2023-11-28 16:04:38 total 0 docs, 0 bytes
2023-11-28 16:04:38 total 0.238 sec, 0 bytes/sec, 0.00 docs/sec
2023-11-28 16:04:38 total 0 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
2023-11-28 16:04:38 total 10 writes, 0.006 sec, 0.1 kb/call avg, 0.6 msec/call avg
2023-11-28 16:04:38 WARNING: failed to open pid_file '/var/run/manticore/searchd.pid'.
2023-11-28 16:04:38 WARNING: tables NOT rotated.

This is my docker-composer.yml file

  db:
    image: "mysql"
    command: --default-authentication-plugin=mysql_native_password --sql_mode="STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"    
    container_name: "mysql-container"
    ports:
      - 3306:3306
    volumes:
      - ./data/mysql/schema.sql:/docker-entrypoint-initdb.d/schema.sql
    environment:
      - MYSQL_ROOT_PASSWORD=password 
      - MYSQL_DATABASE=database
      - MYSQL_USER=user
      - MYSQL_PASSWORD=password
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    command: "gosu manticore indexer --all --rotate"
    restart: always
    depends_on:
      - db
    ports:
      - 9306:9306
      - 9308:9308
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./data/manticore/lib:/var/lib/manticore
      - ./data/manticore/indexes:/etc/manticore/data
      - ./data/manticore/log:/var/log/manticore
      - ./data/manticore/local_cw_manticore.conf:/etc/manticoresearch/manticore.conf 
      - ./data/manticore/stopword/magazine-stopwords.txt:/etc/manticoresearch/magazine-stopwords.txt

I do have this at the top of local_cw_manticore.conf

searchd {
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9307:mysql_vip
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
}

If someone has run into this issue or has some insight into what is happening that would be amazing. We have been running into this issue for the past week now and can’t seem to find a solution to it

2

It seems like you container is just set to run ‘indexer’

command: "gosu manticore indexer --all --rotate"

Where really the command should be running ‘searchd’ - that is the real daemon, that answers queries etc.

When you call ‘indexer --rotate’ it is expecting that searchd is running (and its searchd that creates the pid file, that can’t be found)

Once you’ve got a container running searchd, then figure how to run indexer. Normally would run it with ‘cron’. You could perhaps run cron in the same container as searchd, but it would normally be cleaner to run it in a sidecar container.

The indexer and cron containers, would have to have a shared data volumn, and also be able to communicate (via pid file and signals)

We actully use GitHub - aptible/supercronic: Cron for containers to as the cron daemon in our containerized manticore setup.
So technically would be one container running supercronic (which periodically runs indexer), and the seperate container that runs searchd.

3

indexer has to be run in a running container, please read GitHub - manticoresoftware/docker: Official docker for Manticore Search and use docker exec

Please also read about the new features related with indexing in docker - GitHub - manticoresoftware/docker: Official docker for Manticore Search that allow building a plain table on container startup and using an internal cron service.