Upgrade from 4.0.2 to 4.2 has been failed

General Discussions - English
1

I have been testing an upgrade procedure from (manticore=4.0.2-210921-af497f245, manticore-columnar-lib=1.11.2-210921-69d3780) to version 4.2, unfortunately the process seems to me converting only part of the indices that I have in the cluster, see the log below:

[Sun Jan 30 13:33:55.925 2022] [27678] watchdog: main process 27679 forked ok

[Sun Jan 30 13:33:55.928 2022] [27679] starting daemon version '4.2.0 15e927b28@211223 release (columnar 1.11.4 327b3d4@211223)' ...

[Sun Jan 30 13:33:55.928 2022] [27679] listening on 127.0.0.1:9306 for mysql

[Sun Jan 30 13:33:55.928 2022] [27679] listening on 127.0.0.1:9308 for sphinx and http(s)

[Sun Jan 30 13:33:55.928 2022] [27679] listening on 10.0.82.165:9312 for sphinx and http(s)

[Sun Jan 30 13:33:55.928 2022] [27679] listening on 10.0.82.165:9306 for mysql

[Sun Jan 30 13:33:55.928 2022] [27679] listening on 10.0.82.165:9308 for sphinx and http(s)

[Sun Jan 30 13:33:55.929 2022] [27680] WARNING: index 'my_index1': removed from JSON config

[Sun Jan 30 13:33:55.930 2022] [27680] WARNING: index 'my_index1': prealloc: disk chunk /var/lib/data/manticore/my_index1/my_index1.0: prealloc failed: Unable to load columnar storage: /var/lib/data/manticore/my_index1/my_index1.0.spc is v.5, binary is v.6 - NOT SERVING

[Sun Jan 30 13:33:55.933 2022] [27680] WARNING: index 'my_index': removed from JSON config

[Sun Jan 30 13:33:55.933 2022] [27680] WARNING: index 'my_index': prealloc: disk chunk /var/lib/data/manticore/my_index/my_index.3960: prealloc failed: Unable to load columnar storage: /var/lib/data/manticore/my_index/my_index.3960.spc is v.5, binary is v.6 - NOT SERVING

[Sun Jan 30 13:33:55.989 2022] [27680] WARNING: index 'my_index11': removed from JSON config

[Sun Jan 30 13:33:55.989 2022] [27680] WARNING: index 'my_index11': prealloc: disk chunk /var/lib/data/manticore/my_index11/my_index11.0: prealloc failed: Unable to load columnar storage: /var/lib/data/manticore/my_index11/my_index11.0.spc is v.5, binary is v.6 - NOT SERVING

[Sun Jan 30 13:33:56.027 2022] [27680] WARNING: index 'mc_33a4dec1_workflow_dev_1806': removed from JSON config

[Sun Jan 30 13:33:56.027 2022] [27680] WARNING: index 'mc_33a4dec1_workflow_dev_1806': prealloc: disk chunk /var/lib/data/manticore/mc_33a4dec1_workflow_dev_1806/mc_33a4dec1_workflow_dev_1806.0: prealloc failed: Unable to load columnar storage: /var/lib/data/manticore/mc_33a4dec1_workflow_dev_1806/mc_33a4dec1_workflow_dev_1806.0.spc is v.5, binary is v.6 - NOT SERVING

[Sun Jan 30 13:33:56.029 2022] [27680] WARNING: index 'mc_33a4dec1_workflow_dev_1804': removed from JSON config

[Sun Jan 30 13:33:56.029 2022] [27680] WARNING: index 'mc_33a4dec1_workflow_dev_1804': prealloc: disk chunk /var/lib/data/manticore/mc_33a4dec1_workflow_dev_1804/mc_33a4dec1_workflow_dev_1804.0: prealloc failed: Unable to load columnar storage: /var/lib/data/manticore/mc_33a4dec1_workflow_dev_1804/mc_33a4dec1_workflow_dev_1804.0.spc is v.5, binary is v.6 - NOT SERVING

[Sun Jan 30 13:33:56.065 2022] [27733] prereading 18 indexes

[Sun Jan 30 13:33:56.066 2022] [27736] WARNING: '10.0.82.48:9312': receiving failure (errno=111, msg=Connection refused)

[Sun Jan 30 13:33:56.066 2022] [27736] WARNING: '10.0.82.123:9312': receiving failure (errno=111, msg=Connection refused)

[Sun Jan 30 13:33:56.066 2022] [27736] WARNING: cluster 'CORP_FTS_1': no available nodes (10.0.82.48:9312,10.0.82.123:9312,10.0.82.165:9312), replication is disabled, error: '10.0.82.48:9312': receiving failure (errno=111, msg=Connection refused);'10.0.82.123:9312': receiving failure (errno=111, msg=Connection refused)

[Sun Jan 30 13:33:56.066 2022] [27679] accepting connections

[Sun Jan 30 13:33:56.067 2022] [27733] prereaded 18 indexes in 0.002 sec

Thanks in advance for help,

P

2

Manticore Columnar Library doesn’t provide backward compatibility yet, so you have to rebuild the indexes.

3

@Sergey thx a lot, but how it is possible that some indices were migrated just fine?

4

@Sergey can you guide me how to rebuild the index using Manticore tools?

5

The indexes that migrated fine perhaps didn’t have columnar columns (with engine='columnar'). To rebuild the index you need to just create it from scratch and populate it with your data. If you can’t do it I recommend the following:

  • start the older version
  • dump all from your indexes with help of some script (just SELECT * FROM ... with high max_matches, LIMIT and OFFSET if required) or range by or or by anything else.
  • use SHOW CREATE TABLE to see the indexes schemas
  • backup the indexes
  • remove the indexes
  • run CREATE TABLE
  • use your script to populate the index from your dump
6

Amazing thanks a lot, do you have a script example that helps to dump the index to csv/json and re-import it again?

7

do you have a script example that helps to dump the index to csv/json

Unfortunately not. It would be great if it could be done via mysqldump, but it doesn’t work with Manticore yet. But you should be able to implement such a script in any programming language in 10-20 lines.

8

Thx in case I’ll find some time I’ll contribute to the community, do you prefer PR in here GitHub - manticoresoftware/manticoresearch: Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon?

9

If it’s a Manticore Search improvement which would make it compatible with mysqldump - yes.
If it’s just an external script - some new repo would make more sense.

1 Like
10

@Sergey let’s consider the first option first, how can I dump data from the index before an upgrade if the data was only indexed without storing a full text of original document?

11

There’s no way to restore the original contents of the full-text fields then. It’s stored by default (in RT mode) for a purpose :slight_smile:

12

@Sergey yes I understand, we changed the mode to store only an indexed version of documents to save some disk space and memory. BTW what’s the usual fraction between the size of indexed data and the size of the whole document? Should I consider that as 2x?

13

It’s controlled by https://manual.manticoresearch.com/Creating_an_index/Local_indexes/Plain_and_real-time_index_settings#docstore_compression_level. Using it you can balance between disk space and indexation/search performance (when you touch stored full-text fields).

1 Like