"Right" way to run indexer, searchd

General Discussions - English
1

I am new to Manticore but the more I try to get started with it the more confused I get. I am running Ubuntu 22.04.1 LTS. How should indexer and searchd be run to provide consistent results that do not give errors?

The documentation says that searchd is started by the init system under the user named manticore and all files created by the server will be owned by this user, and that if searchd is started under, for example, the root user, the permissions of files will be changed which may lead to issues when running again searchd as service.

However, the “indexing data from MySQL” tutorial appears to be running under the root user to create indexes and to start searchd. Is this the right way to do things?

If I try running indexer --all as my regular (non-root, non-manticore) username, I get the error

FATAL: failed to open /var/lib/manticore/indx.spl: Permission denied, will not index. Try --rotate option.

If I try running indexer --all as user manticore with the command sudo runuser -l manticore -c 'indexer --all', I get the error

runuser: warning: cannot change directory to /home/manticore: No such file or directory

I can build the indexes as the root user using sudo index --all, but then when I go into MySQL using mysql -P9306 -h0 and run the command SHOW TABLES; it just says Empty set (0.000 sec). Also, if I run ls -ls in /var/lib/manticore, it shows that indx.spa and indx.spb are owned by root:root, and that might be bad, right?

I once was able to get SHOW TABLES; to show the indexes, but I’m not sure what I did to make it work. I think it involved running searchd as root (sudo searchd) first. Now, if I try that, I get the error message

FATAL: failed to lock pid file '/var/run/manticore/searchd.pid': Resource temporarily unavailable (searchd already running?)

So I get the feeling that the interactive tutorials are taking some shortcuts by running as root that I should not be taking. What is the right way to set things up so that my indexes get built, I have no permissions problems, and I can access the indexes using the PHP client manticoresearch-php?

(Trying to run the PHP client to execute a search gives the following error:
PHP Fatal error: Uncaught Manticoresearch\\Exceptions\\ResponseException: "unknown local index(es) 'indx' in search request" in /home/myusername/public_html/vendor/manticoresoftware/manticoresearch-php/src/Manticoresearch/Transport/Http.php:127\nStack trace:\n#0)

2 Likes
2

I found this post which indicated that if the indexes are built after starting searchd, one of the following methods could be used to make the indexed appear to searchd:

  • rebuild the index with indexer --rotate
  • call RELOAD INDEXES via the sql interface
  • restart the searchd

I don’t think I’m able to do the first of these for the reasons mentioned in the post above, and I thought that I tried the third of these option to no effect (using sudo systemctl restart manticore), but the second option, lo and behold, brought up the indexes for the SHOW TABLES; command after entering MySQL using mysql -h0 -P9306.

Now my simple PHP search script works.

I still have a few items on my wishlist, though:

  1. I wish the above solution was better documented in the main documentation.
  2. I hope that the discrepancy between the main documentation and the interactive tutorials can be resolved regarding the use of root and manticore users to perform index building and to start searchd.
  3. I could still use some guidance on the overarching question of what the “right” way to build indexes and start searchd is, so I don’t have any problems in the future.

Thanks!

1 Like
3

the plain flow is

  • define source and index in config
  • create actual indexes with indexer --all
  • start daemon to work with existed indexes via systemctl start manticore

all should be done under manticore user as in case you try one of the operations with sudo indexer or daemon can not access index files or logs or config

you could also stick to RT mode and completely drop indexer but create and populate RT indexes via daemon.

1 Like
4

you could read about RT mode at our Real-time-mode-vs-plain-mode and play with it at our course RTmode

1 Like
5

Actually not. We’ll update the course to make things less confusing. We thought that this part in the documentation is visible enough:

image
image2946Ă—1650 407 KB

but apparently it’s not.

What is the right way to set things up so that my indexes get built, I have no permissions problems, and I can access the indexes using the PHP client manticoresearch-php?

  1. update /etc/manticoresearch/manticore.conf, e.g. like this:
source src {
    type = csvpipe
    csvpipe_command = echo "1,abc" && echo "2,abc" && echo "3,abc abc"
    csvpipe_field = f
}

index idx {
    type = plain
    source = src
    path = /var/lib/manticore/idx
    stored_fields = f
}

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9308:http
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
}
  1. Make sure the path is accessible. Make sure to remove data_dir to switch to the plain mode
  2. systemctl restart manticore
  3. sudo -u manticore indexer ... --rotate , e.g.:
[root@1f128e600e81 /]# sudo -u manticore indexer --all
Manticore 6.0.0 8de9df201@230206 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

using config file '/etc/manticoresearch/manticore.conf'...
indexing table 'idx'...
collected 3 docs, 0.0 MB
creating secondary index
creating lookup: 0.0 Kdocs, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 3 docs, 13 bytes
total 0.042 sec, 302 bytes/sec, 69.90 docs/sec
total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
rotating tables: successfully sent SIGHUP to searchd (pid=222).
  1. mysql -P9306 -h0 -e "reload indexes" since the running server doesn’t yet know about the index, since it didn’t exist on start. Once it knows it, you can do just indexer ... --rotate w/o reload indexes.

After that the table should be accessible:

[root@1f128e600e81 /]# mysql -P9306 -h0 -e "show tables"
+-------+-------+
| Index | Type  |
+-------+-------+
| idx   | local |
+-------+-------+
1 Like
6

Thank you tomat and especially Sergey for the very thorough replies but moreso for taking to heart my suggestion that the interactive course may cause confusion to new users inasmuch as it appears to contradict the documented suggestion not to do things as root.

1 Like