Why expand_keywords as index option not functional


#1

I have an odd issue on a Manticore 2.6.2 0bbd194@180223 server. SOMETIMES the expand_keywords=1 in index, is ignored. not reported as a bug yet, as can’t create a reproducible test case.

sphinxQL>set profiling=1;
Query OK, 0 rows affected (0.00 sec)

sphinxQL>select * from tags where match('str') limit 0; show plan;
Empty set (0.00 sec)

+------------------+-------------------------------+
| Variable         | Value                         |
+------------------+-------------------------------+
| transformed_tree | AND(KEYWORD(str, querypos=1)) |
+------------------+-------------------------------+
1 row in set (0.00 sec)

… yet the expand_keywords = 1 on the index. I have also tried expand_keywords = star

If copy the config to test index, get expansion by default

sphinxQL>select * from test2 where match('str') limit 0; show plan;
Empty set (0.00 sec)

+------------------+-------------------------------------------------------------------------+
| Variable         | Value                                                                   |
+------------------+-------------------------------------------------------------------------+
| transformed_tree | OR(
  AND(KEYWORD(str, querypos=1)),
  AND(KEYWORD(str*, querypos=1))) |
+------------------+-------------------------------------------------------------------------+
1 row in set (0.00 sec)

Note I CAN enable it at query time, but want it for every query on this particular index

sphinxQL>select * from tags where match('str') limit 0 option expand_keywords=1; show plan;
Empty set (0.01 sec)

+------------------+-------------------------------------------------------------------------+
| Variable         | Value                                                                   |
+------------------+-------------------------------------------------------------------------+
| transformed_tree | OR(
  AND(KEYWORD(str, querypos=1)),
  AND(KEYWORD(str*, querypos=1))) |
+------------------+-------------------------------------------------------------------------+
1 row in set (0.00 sec)

diff /etc/sphinxsearch/sphinx.d/test2.conf /etc/sphinxsearch/sphinx.d/tags.conf -y
source srctest2 : db_master                                   | source srctags : db_slave
{                                                               {
                                                              >         sql_query_range         = SELECT MIN(tag_id),MAX(tag_
                                                              >         sql_range_step          = 10000
        sql_query               = \                                     sql_query               = \
                (SELECT 1 as id, 'str' as text) UNION (SELECT |                 SELECT tag_id,prefix,tag,`count` AS images, '
}                                                             |                         GROUP_CONCAT(DISTINCT ts.user_id) AS
                                                              >                         GROUP_CONCAT(DISTINCT ts.grid_referen
                                                              >                         IF(canonical = 0 OR canonical IS NULL
                                                              >                         IF(canonical = 0 OR canonical IS NULL
                                                              >                 FROM tag t INNER JOIN tag_stat USING (tag_id)
                                                              >                         LEFT JOIN tag_square_stat ts USING (t
                                                              >                 WHERE (t.tag_id>=$start AND t.tag_id<=$end) \
                                                              >                 AND t.prefix != 'type' \
                                                              >                 GROUP BY tag_id \
                                                              >                 ORDER BY null
                                                              >         sql_attr_uint           = images
                                                              >         sql_attr_multi          = uint user_id from field;
                                                              >         sql_attr_uint           = grouping
                                                              >         sql_attr_bool           = prefered
                                                              >         sql_field_string        = prefix
                                                              >         sql_field_string        = tag

index test2                                                   | }
                                                              > index tags
{                                                               {
        source                  = srctest2                    |         source                  = srctags
        path                    = /var/lib/manticore/data/tes |         path                    = /var/lib/manticore/data/tag
        dict = crc                                                      dict = crc
        min_word_len            = 1                                     min_word_len            = 1

        min_prefix_len          = 2                                     min_prefix_len          = 2
        min_infix_len           = 0                                     min_infix_len           = 0
        expand_keywords = 1                                   |         expand_keywords = star
}                                                               }

Can anyone spot anything I’ve missed?


#2

Tried to reproduce, but no luck. Did you restarted daemon after changing expand_keywords in config or reindexed?


#3

Haven’t actully restarted, just reloaded. As using “indexer tags --rotate”. In fact expand_keywords = 1 has been in the config for months, and the index is rebuilt every 2 hours via cron anyway.

WHen I said ‘sometimes’, ti will even sometimes work on on the tags index

sphinxQL>set profiling=1;select * from tags where match('str') limit 0; show plan;
Query OK, 0 rows affected (0.00 sec)

Empty set (0.01 sec)

+------------------+-------------------------------------------------------------------------+
| Variable         | Value                                                                   |
+------------------+-------------------------------------------------------------------------+
| transformed_tree | OR(
  AND(KEYWORD(str, querypos=1)),
  AND(KEYWORD(str*, querypos=1))) |
+------------------+-------------------------------------------------------------------------+
1 row in set (0.00 sec)

… but then the index will rebuild via cron, and its turned off again.

sadly indextool --dumpconfig nor --dumpheader seem to report expand_keywords so can’t see if it is or isnt written to the .sph file easily.


#4

Its bit long winded, but it seems repeatable…

As far as can see, its working for exactly one rotation after changing config, then stops working again.

All am doing when changing config is changing 1 <=> star which should both work (and does, but once)


#5

Also seems that restarting manticore makes it work (with no change to the index, nor config!).
… and then will work after the first reindex. But after the second, fails again. no change to config.

I can only guess its something to to with the dynamic config file. sphinx.conf is actully a PHP script, that jsut concats all the indvidiual config files (its a php as it replaces some secret variables from a config file)
… after if not changing config, the output of the PHP doesnt change, and hence may cause searchd to no reload so fully. (iirc searchd uses hash of output in case of dynamic config, rather than timestamp of file if a static file)


#6

Ok, last reply for now. See seems if setup the config file to be ever changing (output a timestamp ever time), then expand_keywords remains functional after many rotations

… not ideal, as will cause searchd to allways reload the whole config, even if nothing changed, but for now, it at least means expand_keywords works :slight_smile:

Will try to build a smaller test case.


#7

Have got a clear reproduction case

seems critial that the config file be a ‘dynamic’ version. Just adding #!/usr/bin/php as first line of the config file triggers the issue.