Data sync from PostgreSQL to RT tables

General Discussions - English
1

I want to sync the data from PostgreSQL database table to manticore RT table by using source command. i tried the below commands and getting source: not found. Can you please help here.

$ cat /etc/manticoresearch/manticore.conf                  
#!/bin/sh
sed 's/$hostname/'$(hostname -f)'/' $CONFIGMAP_PATH

source my_pgsql
{
    type = pgsql
    sql_host = xxxxx.postgres.database.azure.com
    sql_user = read_user
    sql_pass = xxxxx123
    sql_db = newapps
    sql_port = 5432

    sql_query = SELECT id,summary,searchable,rich_text,notes,object_id,updatedat,createdat,ticket_id,tenant_id,submittedby,source,team,itemtypeid,raw_data,data,queues,actors,allactors,status,towatchers,ccwatchers,watchers,tokens FROM testschema.manticore_pgsql WHERE updatedat>1710781130489;
}
$ whoami
manticore
$ source case_values_pgsql_test from my_pgsql;
sh: 5: source: not found
2

I do not know about the source command but if you want get used to indexing plain indexes with data source defined at the config file you could try out interactive course Indexing data from MySQL example I sure PostgreSQL data source works the same way

3

A ‘source’ definition in manticore.conf, is for for creating a ‘plain’ index.

the indexer command is what is used to create the index. Would need a ‘index’ definition to turn into a usable index.

indexer would be used to create the plain index. After creation, could do a one-off conversion to a RT index.

If want to do a ongoing conversion (keeping data in sync) it more complex. Theotetically possible, but the benefits of using a RT index are mostly mute if going to just load data with indexer anyway.

So in general would use indexer to ‘bulk’ build the index when commissioning. To keep it ‘sync’ later, the application would inject data into the RT index on an ongoing basis (ie when it update PSQL database, it also updates manticore table/index. )