Understanding config files, newbie

Hello,

Don’t know if I got this right:

You can only import data using the importer from sql db if the table is plain?

And to have it real- time you have to create a dump delete related parts in the config file, add dada_dir, switch to real-time, import the data to a newly created real time table?
Cause attach wouldn’t work since the table has been deleted from config.

Or start rt, create an rt table and use something else to transfer the data from sql to a newly created rt table (even though manticore has an importer, you can’t use it)

Or stay in plain mode use the config, import the data, attach to a rt table, delete the relevant part in config to stay in rt mode?

Hello

You can only import data using the importer from sql db if the table is plain?

Yes, indexer. And not just from an SQL database — you can also import data from a CSV file, an XML output from a script, or any non-SQL database via ODBC.

And to have it real- time you have to create a dump delete related parts in the config file, add dada_dir, switch to real-time, import the data to a newly created real time table?

If your goal is to first index data from an SQL database using indexer and then switch to real-time mode, here’s what you can do:

  • Create a separate configuration file and include the following:
    • A plain table with a source to index from the database.
    • A real-time table with the same schema as in the plain table.
    • Use a different port (e.g., listen = localhost:9307:mysql), pid_file, and log path. You can set binlog_path = # to disable binary logging.
  • Run indexer.
  • Start this separate instance.
  • Use ATTACH to link the plain table to the real-time table.
  • In your main instance, use IMPORT to bring data from the real-time table in the first instance.

Remember to run indexer and searchd under user manticore to avoid permission issues.

Thanks for your answer and time, my goal was to have a realtime table but with initial data populated from existing db/file/dataset but also be able to recreate/reindex everything if needed,

One would act as realtime db search(manticore) and autocomplete with fuzzy ( although there are limitations there too) and the other db/file as bkp but wanted to be able to recreate the manticore tables. indexes on the fly.

Define table(preferably realtime) in config, run importer to populate data from whatever (sql, file, etc), resume activity and update/insert/delete records as needed but have them instantly available for search.

Plain table I have to re-index it with every update or insertion

Guess the approach would be to have another tool that creates the initial real-time table according to the schema in sql/file and inserts the initial data.

Unfortunately, indexer does not support real-time tables yet. In your case, I would suggest writing a script to read from your data source and insert the data into Manticore in batches with perhaps some concurrency. This approach could even be faster than using indexer. Typically, this type of script is around 20-30 lines of code (without concurrency). Adding concurrency might make it slightly more complex (for example, here is one we use in our tests: manticoresearch/test/clt-tests/scripts/load_us_names_min_infix_len.php at 22c2a280199694157b98a9f4b9a3b4998e107ab5 · manticoresoftware/manticoresearch · GitHub).

To re-index, you can use mysqldump -etc --replace as described here: Manticore Search Manual: Securing and compacting a table > Backup and restore. For example:

mysqldump -P9306 -h0 -etc --replace manticore table_name | mysql -P9306 -h0

This command will re-index the table_name by re-inserting all documents into it.

1 Like