RT and Plain Indexes

General Discussions - English
1

In the documentation it says that you cannot mix RT with Plain indexes which is why I’m getting the error: FATAL: 'data_dir' cannot be mixed with index declarations in '/etc/manticoresearch/manticore.conf'

Where do I create the plain index(es) for my CSV file(s) so that I can eventually attach them to my RT index? When I put the source and index declarations in the same manticore.conf file, I get the error above. Do I need to literally duplicate the manticore file and set different values for data_dir, logs, ports, etc… in searchd and have a different config for the plain index(es)?

2

you can not create or use plain index in Real-time mode. However you could follow the description to get your data.

To use your already build plain index in Real-time mode you have

  • create plain in RT index in plain mode at config (plain.conf)
  • start daemon with config (plain.conf) in plain mode
  • issue ATTACH statement to attach plain index into RT index
  • shutdown daemon
  • start daemon with real-time mode
  • issue IMPORT statement to import your RT index from some directory

This way you will get your plain index into Real-time mode.

3

Is there a way to do this without having to shutdown the daemon? Maybe running on separate servers or possibly running the plain-mode version in a docker instance?

4

you need 2 daemons

  • first with plain-mode config that attaches plain into RT
  • second with RT-mode config that imports RT index

you could try any sequence that you want

5

Just to try it out, I copied/pasted default manticore config file and added the source/index for my plain index. The docker instance shuts down with an exit 0. I have to remove the source/index from the config in order to get it to run properly. I understand that RT-Mode is the default, I don’t see any settings that allow you to switch from RT to Plain.

6

You’d better just maintain 2 configs:

  1. for indexation
    • just for building an index (you can have just source and index there)
    • if you also want to prepare an RT index from the plain index with help of ATTACH you also need searchd since you’ll have to run searchd for that
  2. just a normal config in RT mode (i.e. with data_dir)

It’s ok that docker exec ... indexer exits with 0.

7

Can I attach it from the RT index without having to add a searchd to the plain config? or does the attach have to happen from the plain config?

I tried to run the following:
mysql> attach index sample to rtindex parts; ERROR 1064 (42000): no such index 'sample' mysql>

The data exists. I have a directory in my data directory named “sample”, which is what I named it and it looks like it was indexed as expected.

8

import works only with rt index

rt-mode where import works does not work with plain indexes

I already posted links into manual with detailed explanation

9

I’m upping this topic because I stuck with the same situation and as I see documentation and anwsers in this topic doesn’t make it clear.

We got into situation where we need to reindex 35M records. Our tool we are using to occasionally update index got stuck in such ammount (it reads about 1000 records from DB and updates index by id via REPLACE). So I decided to go through creating plain index at first (it is faster - takes about 2 1-1,5 hours to create) and attach to rt-index with truncate. We did this some years earlier (on v4 of Manticore as I remember).

Nowadays I wanted to update this for the latest v6.3 and wanted to make it using “best practices”. As these practices suggest - it is better to have separate configs for plain mode and rt mode. We have a “mixed” config for now (with rt-index but no data_dir defined in config) and as I found out it is called plain mode (correct me if I’m wrong).

So I created plain.conf (with source and index[type=plain] only defined). docker run ... indexer --all successfully creates plain index in my volume.

Next step as suggested earlier in this thread (and documentation) states that at first I need to ATTACH plain table to rt table before I can IMPORT it. But as hatemjaber wrote, I got an error ERROR 1064 (42000): no such table.

So my question is - Do I need to define RT-index in plain.conf also? Because my RT-indicies are defined in rt.conf

10

yes

for config not-RT mode:

  • need plain index definition and RT index definition in the config
  • after indexing you need to rotate plain index into the daemon
  • issue ATTACH plain table TO rt table WITH TRUNCATE

only after steps you will have RT index with the content you indexed into the plain index that could be used for IMPORT at daemon in the RT mode