Error with csvpip

General Discussions - English
1

Hello

I hav tried during hours to solve this bug :
ERROR: index ‘idx_ca’: source ‘src_ca’: not all columns found (found=57, total=56, line=71062)
I hav checked my csv no problem with number of values per line.

I think is a bug in manticore.
Any solution or advices ?

Here is a sample of my csv :
tsv: https://62-210-162-79.rev.poneytelecom.eu/test.txt
tsv with header : https://62-210-162-79.rev.poneytelecom.eu/test_with_header.txt
and my conf
source src_ca
{
type = csvpipe
csvpipe_command = cat /root/INSEE_CA_RE_OK_AVEC_NUMCA_TAB.tsv
csvpipe_delimiter= ¤

csvpipe_field_string =  NOM
csvpipe_field_string =  SIREN
csvpipe_field_string =  SIRET
csvpipe_field_string =  APE
csvpipe_field_string =  FORME
csvpipe_field_string =  SECTEUR
csvpipe_field_string =  VILLE
csvpipe_field_string =  CP
csvpipe_field_string =  DEP
csvpipe_field_string =  REG
csvpipe_field_string =  GREFFE
csvpipe_field_string =  ADRESSE
csvpipe_field_string =  DATE_CREA
csvpipe_field_string =  ANNEE_CREA
csvpipe_attr_float =  EFF_2011
csvpipe_attr_float =  CA_2011
csvpipe_attr_float =  RE_2011
csvpipe_attr_float =  DUREE_EXE_2011
csvpipe_attr_float =  EFF_2012
csvpipe_attr_float =  CA_2012
csvpipe_attr_float =  RE_2012
csvpipe_attr_float =  DUREE_EXE_2012
csvpipe_attr_float =  EFF_2013
csvpipe_attr_float =  CA_2013
csvpipe_attr_float =  RE_2013
csvpipe_attr_float =  DUREE_EXE_2013
csvpipe_attr_float =  EFF_2014
csvpipe_attr_float =  CA_2014
csvpipe_attr_float =  RE_2014
csvpipe_attr_float =  DUREE_EXE_2014
csvpipe_attr_float =  EFF_2015
csvpipe_attr_float =  CA_2015
csvpipe_attr_float =  RE_2015
csvpipe_attr_float =  DUREE_EXE_2015
csvpipe_attr_float =  EFF_2016
csvpipe_attr_float =  CA_2016
csvpipe_attr_float =  RE_2016
csvpipe_attr_float =  DUREE_EXE_2016
csvpipe_attr_float =  EFF_2017
csvpipe_attr_float =  CA_2017
csvpipe_attr_float =  RE_2017
csvpipe_attr_float =  DUREE_EXE_2017
csvpipe_attr_float =  EFF_2018
csvpipe_attr_float =  CA_2018
csvpipe_attr_float =  RE_2018
csvpipe_attr_float =  DUREE_EXE_2018
csvpipe_attr_float =  EFF_2019
csvpipe_attr_float =  CA_2019
csvpipe_attr_float =  RE_2019
csvpipe_attr_float =  DUREE_EXE_2019
csvpipe_attr_float =  EFF_2020
csvpipe_attr_float =  CA_2020
csvpipe_attr_float =  RE_2020
csvpipe_attr_float =  DUREE_EXE_2020
csvpipe_attr_float =  NUM_CA
csvpipe_field_string =  LIST_ANNEE

}

2

could you create ticket at Github there to put config and csv files that cause the issue?

3

Thanks for the clue but im not familiar with github.
Maybe someone who have sucessfully used csv/tsvpipe to create an index can share with me ?

4

I hav fixed the problem with csv import, in fact i just forgot to add the id field in my header and data.
But i have en other error, sphinxQl says (http or mysql client ) : Error SQL (1064) : unknown local index(es). Even if the indexer says averything is fine after indexing and rotating the index created from csvpipe.

Any solution ?

5

could you show SphinxQL show table output?

6

I did that
No tables showed

7

when you should look at your daemon searchd.log to check for errors or restart daemon with —logdebug command line option to get more info on index loading process

8

Thank you, i checked the log file , below details :

[Sat Aug 29 23:11:27.314 2020] [10483] WARNING: index ‘idx_ca’: rotating index ‘idx_ca’: .new prealloc failed: failed to open file ‘/var/lib/manticore/data/idx_ca.spa’: ‘Permission denied’; NOT SERVING
[Sat Aug 29 23:11:27.314 2020] [10483] WARNING: index ‘idx_ca’: prealloc: failed to open file ‘/var/lib/manticore/data/idx_ca.spa’: ‘Permission denied’ - NOT SERVING
[Sat Aug 29 23:11:27.314 2020] [10487] prereading 0 indexes
[Sat Aug 29 23:11:27.314 2020] [10487] prereaded 0 indexes in 0.000 sec
[Sat Aug 29 23:11:27.314 2020] [10481] accepting connections
[Sat Aug 29 23:13:32.895 2020] [10481] caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
[Sat Aug 29 23:13:32.895 2020] [10485] WARNING: nothing to rotate after SIGHUP

9

seems indexer and daemon run with different user privileges or indexer runs with sudo that is why daemon failed to open index files

10

i use root in ssh, how i can fix this ?

11

you must chown index files to user that runs daemon

12

I hav done that for every folder where manticore is installed. In fact every time i want to create a new plain index or rotate i hav a problem with folder or pid file already run by another process.
Is there a comman i can use to say to manticore that everything is fine with folders and pid file ?

13

Make sure you run the searchd and the indexer under the same user.

14

It is : i connect to server in ssh with root acces
I use indexer command when errors with pid file or permission
I try search commands.
I dont understand the problem.

15

without looking at your commands it is not clear what to suggest.
Could you dump you terminal output?

16
  • Show ps aux|grep searchd
  • How do you run Manticore search instance?
  • How do you run indexer?

I suspect you do systemctl start manticore which by defaults run searchd not under root. Therefore when you then run indexer under root it has enough rights to write the new index files, but when you then try to use it with the Manticore Search instance it doesn’t have right as it’s not run under root. It’s just my assumption.

17

When i do “ps aux|grep searchd”
i have this
mantico+ 4820 0.0 0.0 38536 3896 ? S Aug30 0:00 /usr/bin/search d --config /etc/manticoresearch/manticore.conf
mantico+ 4821 0.0 0.0 827072 7936 ? Sl Aug30 0:14 /usr/bin/search d --config /etc/manticoresearch/manticore.conf
root 24384 0.0 0.0 12716 1020 pts/0 S+ 11:11 0:00 grep searchd

I have tried this 3 commands
systemctl start manticore
searchd -c /etc/manticoresearch/manticore.conf
/etc/init.d/manticore start

When i do systemctl or /etc/init my console seems tyo be in a dead loop with “[…] Starting manticore (via systemctl): manticore.service” i have to do CRTL+Z

My system is debian9 32GB of ram

Is there a proper way to index my tables without thes pid and folder permission ?

Last lines of my log file :

[Sun Aug 30 22:36:24.969 2020] [4822] WARNING: rotating index ‘idx_ca’: .new preload failed: failed to open file ‘/var/lib/manticore/data/idx_ca.spa’: ‘Permission denied’
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.sph’, error ‘No such file or directory’
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spa’, error ‘No such file or directory’
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spi’, error ‘No such file or directory’
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spd’, error ‘No such file or directory’
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spp’, error ‘No such file or directory’
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spe’, error ‘No such file or directory’
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spm’, error ‘No such file or directory’
[Sun Aug 30 22:36:24.983 2020] [4822] rotating index ‘idx_ca’: success
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: index ‘idx_ca’: rotating index ‘idx_ca’: .new preload failed; ROLLBACK FAILED; INDEX UNUSABLE
[Sun Aug 30 22:36:24.983 2020] [4822] WARNING: index ‘idx_ca’: prealloc: failed to open file ‘/var/lib/manticore/data/idx_ca.spa’: ‘Permission denied’ - NOT SERVING
[Sun Aug 30 22:36:24.991 2020] [4821] accepting connections
[Sun Aug 30 22:36:24.991 2020] [4826] prereading 0 indexes
[Sun Aug 30 22:36:24.991 2020] [4826] prereaded 0 indexes in 0.000 sec
[Sun Aug 30 22:37:31.851 2020] [4821] caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
[Sun Aug 30 22:37:31.851 2020] [4823] WARNING: nothing to rotate after SIGHUP
[Sun Aug 30 22:39:18.342 2020] [4821] caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
[Sun Aug 30 22:39:18.342 2020] [4823] Config changed (read 25693 chars)
[Sun Aug 30 22:39:18.342 2020] [4823] rotating index ‘idx_ca’: started
[Sun Aug 30 22:39:18.343 2020] [4823] WARNING: index ‘idx_ca’: prealloc: failed to open file ‘/var/lib/manticore/data/idx_ca.new.spa’: ‘Permission denied’
[Sun Aug 30 22:39:18.343 2020] [4823] rotating index: all indexes done
[Mon Aug 31 11:19:37.276 2020] [24559] watchdog: main process 24560 forked ok
[Mon Aug 31 11:19:37.277 2020] [24560] FATAL: failed to lock pid file ‘/var/run/manticore/searchd_v2.pid’: Resource temporarily unavailable (searchd already running?)
[Mon Aug 31 11:19:37.278 2020] [24559] watchdog: main process 24560 exited cleanly (exit code 1), shutting down
[Mon Aug 31 11:19:37.278 2020] [24559] watchdog: got error 10, No child processes

18

that error means that index files belong to another user - maybe root user? That is why daemon can not access these files and uses indexes. You have to figure that out and change index file permissions to manticore user

means that another daemon already running and uses the same config and pid file or you run daemon previously from root user and now pid file of daemon also belongs to root user - you have to figure that out and change log \ pid and other daemon related files permissions to manticore user

19

i have tried to change folder permission
i hav tried searchd -c /etc/manticoresearch/manticore.conf command
error : failed to open pid_file ‘/var/run/manticore/searchd_v2.pid’.

and indexer --rotate --all
error in log :

[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: rotating index ‘idx_ca’: .new preload failed: failed to open file ‘/var/lib/manticore/data/idx_ca.spa’: ‘Permission denied’
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.sph’, error ‘No such file or directory’
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spa’, error ‘No such file or directory’
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spi’, error ‘No such file or directory’
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spd’, error ‘No such file or directory’
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spp’, error ‘No such file or directory’
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spe’, error ‘No such file or directory’
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: unlink failed (file ‘/var/lib/manticore/data/idx_ca.old.spm’, error ‘No such file or directory’
[Mon Aug 31 13:44:47.166 2020] [27159] rotating index ‘idx_ca’: success
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: index ‘idx_ca’: rotating index ‘idx_ca’: .new preload failed; ROLLBACK FAILED; INDEX UNUSABLE
[Mon Aug 31 13:44:47.166 2020] [27159] WARNING: index ‘idx_ca’: prealloc: failed to open file ‘/var/lib/manticore/data/idx_ca.spa’: ‘Permission denied’ - NOT SERVING
[Mon Aug 31 13:44:47.167 2020] [27162] prereading 0 indexes
[Mon Aug 31 13:44:47.167 2020] [27162] prereaded 0 indexes in 0.000 sec
[Mon Aug 31 13:44:47.167 2020] [27157] accepting connections
[Mon Aug 31 13:46:04.708 2020] [27157] caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
[Mon Aug 31 13:46:04.708 2020] [27158] WARNING: nothing to rotate after SIGHUP

It seems an infinite problem, a solution send to another problem.

Is there a process to follow in order to allow manticore to access pid and folder ?

Thanks a lot for your help.

20

you need to stop your daemon then issue chown command to folder and to file then check with ls the permissions set and should then you can start your usual command and with ls find what command change permissions