Can I use a custom dict library file when option morphology = 'icu_chinese'

General Discussions - English
1

For example:

dict_path=/your_path/my_dict.txt

/your_path/my_dict.txt

泸州
资阳

Words like ‘泸州’ and ‘资阳’ are names of Chinese cities， But the query for these words is not recognized as words

MySQL [(none)]> select id from wecom_customer where match('资阳')
MySQL [(none)]> show meta;          
+----------------+------------------------------+
| Variable_name  | Value                        |
+----------------+------------------------------+
| total          | 6                            |
| total_found    | 6                            |
| total_relation | eq                           |
| time           | 0.002                        |
| keyword[0]     | 资                           |
| docs[0]        | 147                          |
| hits[0]        | 180                          |
| keyword[1]     | 阳                           |
| docs[1]        | 5360                         |
| hits[1]        | 7825                         |
| index          | corpid:SecondaryIndex (100%) |
+----------------+------------------------------+
11 rows in set (0.00 sec)

MySQL [(none)]> show create table wecom_customer;
CREATE TABLE wecom_customer (
id bigint,
province string indexed attribute,
city string indexed attribute,
...
deletetime timestamp
) charset_table='0..9, english, chinese' morphology='icu_chinese' stopwords='/home/sphinx/expo/wecom_customer/zh'

Is there a configuration option for manticore like this?

2

I would expect Manticore Search Manual: Creating a table > NLP and tokenization > Exceptions do the trick, but it doesn’t:

➜  ~ cat << 'EOF' > /tmp/exc
泸州 => 泸州
资阳 => 资阳
EOF

mysql -v -P9306 -h0 -e "drop table if exists t; create table t(f text) morphology='icu_chinese' charset_table='cjk' exceptions='/tmp/exc'; insert into t values(1, '泸州是四川省的一座历史文化名城，以其浓郁 的酒文化而闻名'), (2, '资阳市位于四川东南部，是一个快速发展的现代化城市'); select * from t where match('资阳'); show meta; call keywords('资阳', 't')"
--------------
drop table if exists t
--------------

--------------
create table t(f text) morphology='icu_chinese' charset_table='cjk' exceptions='/tmp/exc'
--------------

--------------
insert into t values(1, '泸州是四川省的一座历史文化名城，以其浓郁的酒文化而闻名'), (2, '资阳市位于四川东南部，是一个快速发展的现代化城市')
--------------

--------------
select * from t where match('资阳')
--------------

+------+--------------------------------------------------------------------------+
| id   | f                                                                        |
+------+--------------------------------------------------------------------------+
|    2 | 资阳市位于四川东南部，是一个快速发展的现代化城市                         |
+------+--------------------------------------------------------------------------+
--------------
show meta
--------------

+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| total          | 1     |
| total_found    | 1     |
| total_relation | eq    |
| time           | 0.000 |
| keyword[0]     | 资    |
| docs[0]        | 1     |
| hits[0]        | 1     |
| keyword[1]     | 阳    |
| docs[1]        | 1     |
| hits[1]        | 1     |
+----------------+-------+
--------------
call keywords('资阳', 't')
--------------

+------+-----------+------------+
| qpos | tokenized | normalized |
+------+-----------+------------+
| 1    | 资        | 资         |
| 2    | 阳        | 阳         |
+------+-----------+------------+

Feel free to file it as a bug on github.

3

Ok, thanks

4

@stcer 你也要用自定义中文分词吗