Is there any way to make the indexer ignore "WARNING: word overrun buffer, clipped!!!"

General Discussions - English
1

Hello,

I noticed many warnings like

WARNING: word overrun buffer, clipped!!!
clipped (len=126, word='𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛

                                                        original (len=128, word='𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛')

This from what I know simply means the length of text without spacing is over 127 is clipped.
However, as our index is pretty large, these warnings are annoying especially when debugging output.

Is there anyway to get the indexer to ignore such warnings and not output them?

Thank you.

2

I too get lots of these, is there a way to change the buffer size ? Does it make sense (if only to avoid the warning messages) ?

3

As a workaround you can do grep -v, e.g.:

cat << 'EOF' > min_long_word.conf
source src {
    type = csvpipe
    csvpipe_command = echo "1,𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛"
    csvpipe_field = f
}

index plain {
    type = plain
    source = src
    path = /tmp/plain
}
EOF

indexer -c min_long_word.conf --all

indexer -c min_long_word.conf --all 2>&1 | egrep -v "word overrun buffer|, word="

gives:

Without egrep -v:

Manticore 13.4.1 3f13afa91@25072215 dev (columnar 8.0.1 fc30df6@25072219) (secondary 8.0.1 fc30df6@25072219) (knn 8.0.1 fc30df6@25072219)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)

using config file '/Users/sn/manticore_github/min_long_word.conf'...
indexing table 'plain'...
WARNING: word overrun buffer, clipped!!!
clipped (len=126, word='𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛�')
original (len=128, word='𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛𐨛')
collected 1 docs, 0.0 MB
creating secondary index
creating lookup: 0.0 Kdocs, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 1 docs, 128 bytes
total 0.023 sec, 5359 bytes/sec, 41.86 docs/sec
total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg

With egrep -v:

Manticore 13.4.1 3f13afa91@25072215 dev (columnar 8.0.1 fc30df6@25072219) (secondary 8.0.1 fc30df6@25072219) (knn 8.0.1 fc30df6@25072219)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)

using config file '/Users/sn/manticore_github/min_long_word.conf'...
indexing table 'plain'...
collected 1 docs, 0.0 MB
creating secondary index
creating lookup: 0.0 Kdocs, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 1 docs, 128 bytes
total 0.016 sec, 7561 bytes/sec, 59.07 docs/sec
total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg

Feel free to create a feature request in Github about indexer --quiet.