Attach plain to rt index and reindex

General Discussions - English
1

Hello everyone, could you please help me solve this issue?

Here’s the setup:

  1. A source from MySQL.
  2. A plain index that pulls data from this source.
  3. An RT index.

The workflow is as follows:

indexer --all --rotate
ATTACH TABLE index_plain TO TABLE index_plain_rt WITH TRUNCATE

Everything works fine, but here’s the question: How can I update the index and perform the ATTACH again?

The problem is that after performing ATTACH, the index_plain disappears, and running indexer --all --rotate again does not recreate it.

The only solution I’ve found so far is:

TRUNCATE RTINDEX index_plain_rt
rm -rf /var/lib/manticore/*
indexer --all --rotate
stop docker container
start docker container
ATTACH TABLE index_plain TO TABLE index_plain_rt WITH TRUNCATE

Is there any other way to achieve this without restarting the Docker container and deleting /var/lib/manticore/*?

2

When you run the ‘indexer --rotate’ I suspect it IS recreating the plain index. The thing is that you indexer finishes creating the new index, it signals searchd to reload. searchd looks though its loaded indexes and see if a new version is available.

The thing is searchd no longer had the plain index loaded, so it doesnt ‘notice’ indexer created a new version.

I think you should just be able to use ‘indexer --all’ to recreate the index. Beause you previously used ATTACH, means the plain index isnt loaded by searchd, hence dont need --rotate.

3

indexer --all didn’t do anything when the plain index was attached to the real-time index, so I can’t reindex and update the information.

4

I can’t reproduce the issue. Here’s a fully reproducible example. If you can modify it, so the issue manifests, please do.

➜  ~ cat min_plain_rt.conf
searchd {
    listen = 9315:mysql41
    log = searchd.log
    pid_file = searchd.pid
    binlog_path =
}

source src {
    type = csvpipe
    csvpipe_command = echo "1,abc"
    csvpipe_field = f
}

index plain {
    type = plain
    source = src
    path = /tmp/plain
}

table rt {
    type = rt
    path = /tmp/idx

    rt_field = f
}

➜  ~ rm -f /tmp/plain.* /tmp/rt.*

➜  ~ searchd -c min_plain_rt.conf
Manticore 6.3.9 c96df395d@24112911 dev (columnar 2.3.1 edadc69@24112219) (secondary 2.3.1 edadc69@24112219) (knn 2.3.1 edadc69@24112219)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

[12:55.405] [631399705] using config file '/Users/sn/min_plain_rt.conf' (335 chars)...
starting daemon version '6.3.9 c96df395d@24112911 dev (columnar 2.3.1 edadc69@24112219) (secondary 2.3.1 edadc69@24112219) (knn 2.3.1 edadc69@24112219)' ...
listening on all interfaces for mysql, port=9315
precaching table 'plain'
Index header format is not json, will try it as binary...
WARNING: Unable to load header... Error failed to open /tmp/plain.sph: No such file or directory
WARNING: table 'plain': prealloc: failed to open /tmp/plain.sph: No such file or directory - NOT SERVING
precaching table 'rt'
precached 1 tables in 0.001 sec

➜  ~ mysql -P9315 -h0 -e "show tables"
+-------+------+
| Table | Type |
+-------+------+
| rt    | rt   |
+-------+------+

➜  ~ indexer -c min_plain_rt.conf --all --rotate
Manticore 6.3.9 c96df395d@24112911 dev (columnar 2.3.1 edadc69@24112219) (secondary 2.3.1 edadc69@24112219) (knn 2.3.1 edadc69@24112219)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

using config file '/Users/sn/min_plain_rt.conf'...
indexing table 'plain'...
collected 1 docs, 0.0 MB
creating secondary index
creating lookup: 0.0 Kdocs, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 1 docs, 3 bytes
total 0.069 sec, 42 bytes/sec, 14.29 docs/sec
WARNING: skipping non-plain table 'rt'...
total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 15 writes, 0.001 sec, 0.0 kb/call avg, 0.1 msec/call avg
rotating tables: successfully sent SIGHUP to searchd (pid=69329).

➜  ~ mysql -P9315 -h0 -e "show tables"
+-------+------+
| Table | Type |
+-------+------+
| rt    | rt   |
+-------+------+
➜  ~ mysql -P9315 -h0 -e "reload tables"
➜  ~ mysql -P9315 -h0 -e "show tables"
+-------+-------+
| Table | Type  |
+-------+-------+
| plain | local |
| rt    | rt    |
+-------+-------+

➜  ~ mysql -P9315 -h0 -e "attach table plain to table rt"
➜  ~ mysql -P9315 -h0 -e "show tables"
+-------+------+
| Table | Type |
+-------+------+
| rt    | rt   |
+-------+------+
➜  ~ mysql -P9315 -h0 -e "select * from rt;"
+------+------+
| id   | f    |
+------+------+
|    1 | abc  |
+------+------+

# change in the config:
# - csvpipe_command = echo "1,abc"
# + csvpipe_command = echo "2,abc"

➜  ~ indexer -c min_plain_rt.conf --all --rotate
Manticore 6.3.9 c96df395d@24112911 dev (columnar 2.3.1 edadc69@24112219) (secondary 2.3.1 edadc69@24112219) (knn 2.3.1 edadc69@24112219)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

using config file '/Users/sn/min_plain_rt.conf'...
indexing table 'plain'...
collected 1 docs, 0.0 MB
creating secondary index
creating lookup: 0.0 Kdocs, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 1 docs, 3 bytes
total 0.065 sec, 45 bytes/sec, 15.23 docs/sec
WARNING: skipping non-plain table 'rt'...
total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
rotating tables: successfully sent SIGHUP to searchd (pid=71003).

➜  ~ mysql -P9315 -h0 -e "show tables"
+-------+-------+
| Table | Type  |
+-------+-------+
| plain | local |
| rt    | rt    |
+-------+-------+

➜  ~ mysql -P9315 -h0 -e "attach table plain to table rt"
➜  ~ mysql -P9315 -h0 -e "select * from rt;"
+------+------+
| id   | f    |
+------+------+
|    1 | abc  |
|    2 | abc  |
+------+------+