indextool check: Error failed to open ...

serhio · May 28, 2024, 12:30pm

Доброго времени суток!
Имеется:

manticore-backup/jammy,now 1.3.8-24052208-57fc406 all [установлен, автоматически]
manticore-buddy/jammy,now 2.3.10-24052208-7612a4f all [установлен, автоматически]
manticore-columnar-lib/jammy,now 2.3.0-24052206-88a01c3 amd64 [установлен, автоматически]
manticore-common/jammy,now 6.3.0-24052209-1811a9efb all [установлен, автоматически]
manticore-dev/jammy,now 6.3.0-24052209-1811a9efb all [установлен, автоматически]
manticore-executor/jammy,now 1.1.6-24052206-c55bc2b amd64 [установлен, автоматически]
manticore-extra/jammy,now 1.1.6-24052206-c55bc2b all [установлен]
manticore-galera/jammy,now 3.37 amd64 [установлен, автоматически]
manticore-icudata-65l/jammy,now 5.0.3-221123-d2d9e5e56 all [установлен, автоматически]
manticore-repo/now 0.0.4 all [установлен, локальный]
manticore-server-core/jammy,now 6.3.0-24052209-1811a9efb amd64 [установлен, автоматически]
manticore-server/jammy,now 6.3.0-24052209-1811a9efb amd64 [установлен, автоматически]
manticore-tools/jammy,now 6.3.0-24052209-1811a9efb amd64 [установлен, автоматически]
manticore-tzdata/jammy,now 1.0.0-240522-a8aa66e all [установлен, автоматически]

В Manticore есть следующие индексы:

+-------------------------------+------+
| Index                         | Type |
+-------------------------------+------+
| bindings                      | rt   |
| categories                    | rt   |
| literature_work_cycle_volumes | rt   |
| literature_work_cycles        | rt   |
| persons                       | rt   |
| product_collections           | rt   |
| product_properties_values     | rt   |
| product_tags                  | rt   |
| products                      | rt   |
| products_for_autocorrect      | rt   |
| publisher_brands              | rt   |
| publisher_series              | rt   |
| publishers                    | rt   |
| retail_products               | rt   |
| school_properties_values      | rt   |
| shops                         | rt   |
| tbks                          | rt   |
+-------------------------------+------+

При выполнении команды:

indextool --check retail_products

Всегда есть ошибки вида:

check FAILED, 1 failures reported, 328.0 sec elapsed                                                                                             
checking disk chunk, extension 6021, 32(33)...                                                                                                   
WARNING: Index header format is not json, will try it as binary...                                                                               
WARNING: Unable to load header... Error failed to open /var/lib/manticore/retail_products/retail_products.6021.sph: No such file or directory    
FAILED, disk chunk /var/lib/manticore/retail_products/retail_products.6021: prealloc failed: failed to open /var/lib/manticore/retail_products/re
tail_products.6021.sph: No such file or directory                                                                                                
check FAILED, 2 failures reported, 328.0 sec elapsed

Сам индекс большой, порядка 32Gb весь каталог.
Вчера его специально пересоздали полностью, но проблему не решило.

Это не позволяет пройти проверки в indextool и снять backup full с помощью manticore-backup.

С чем связаны такие ошибки? Почему пытается открыть файлы, которых нет и как устранить это проблему?

tomat · May 28, 2024, 1:41pm

indextool при проверке RT индекс retail_products пытается загрузить disk chunk оканчивающийся на 6021 и говорит что нет таких файлов.
Странно что в демоне загружен этот индекс и демон не ругается, а indextool ругается.

Лучше бы вывод ls -lh /var/lib/manticore/retail_products/retail_products.*.sph привести

serhio · May 28, 2024, 1:54pm

-rw------- 1 manticore manticore 9,9K мая 28 10:58 /var/lib/manticore/retail_products/retail_products.5704.sph
-rw------- 1 manticore manticore 9,9K мая 28 11:42 /var/lib/manticore/retail_products/retail_products.5908.sph
-rw------- 1 manticore manticore 9,9K мая 28 11:54 /var/lib/manticore/retail_products/retail_products.5953.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:26 /var/lib/manticore/retail_products/retail_products.6066.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:28 /var/lib/manticore/retail_products/retail_products.6067.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:29 /var/lib/manticore/retail_products/retail_products.6074.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:43 /var/lib/manticore/retail_products/retail_products.6122.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:55 /var/lib/manticore/retail_products/retail_products.6167.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:57 /var/lib/manticore/retail_products/retail_products.6168.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:59 /var/lib/manticore/retail_products/retail_products.6171.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:13 /var/lib/manticore/retail_products/retail_products.6221.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:15 /var/lib/manticore/retail_products/retail_products.6223.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:18 /var/lib/manticore/retail_products/retail_products.6225.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:20 /var/lib/manticore/retail_products/retail_products.6227.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:23 /var/lib/manticore/retail_products/retail_products.6238.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:36 /var/lib/manticore/retail_products/retail_products.6303.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:38 /var/lib/manticore/retail_products/retail_products.6304.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:39 /var/lib/manticore/retail_products/retail_products.6305.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:41 /var/lib/manticore/retail_products/retail_products.6306.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:43 /var/lib/manticore/retail_products/retail_products.6307.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:46 /var/lib/manticore/retail_products/retail_products.6319.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:52 /var/lib/manticore/retail_products/retail_products.6353.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:53 /var/lib/manticore/retail_products/retail_products.6354.sph

tomat · May 28, 2024, 1:56pm

можете еще вывод

cat /var/lib/manticore/retail_products/retail_products.meta

serhio · May 28, 2024, 2:11pm

Да конечно:

{
"meta_created_time_utc":"2024-05-28T14:09:30.361",
"meta_version":21,
"total_documents":26832093,
"total_bytes":0,
"tid":0,
"schema":{
"attributes":[
{"name":"id","type":6,"locator":{"pos":0,"bits":64}},
{"name":"$_blob_locator","engine":1,"type":6,"locator":{"pos":64,"bits":64}},
{"name":"image","type":7,"locator":{"pos":0,"bits":0}},
{"name":"marketing_status_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"literature_work_publishing_year","type":1,"locator":{"pos":128,"bits":32}},
{"name":"preorder_available_at","type":2,"locator":{"pos":160,"bits":32}},
{"name":"released_at","type":2,"locator":{"pos":192,"bits":32}},
{"name":"rating_star","type":1,"locator":{"pos":224,"bits":32}},
{"name":"rating_average","type":5,"locator":{"pos":256,"bits":32}},
{"name":"rating_weight","type":1,"locator":{"pos":288,"bits":32}},
{"name":"review_count","type":1,"locator":{"pos":320,"bits":32}},
{"name":"purchase_stats_day_avg_count","type":1,"locator":{"pos":352,"bits":32}},
{"name":"is_school_prepare","type":4,"locator":{"pos":384,"bits":1}},
{"name":"is_out_of_class_reading","type":4,"locator":{"pos":385,"bits":1}},
{"name":"school_purpose_id","type":1,"locator":{"pos":416,"bits":32}},
{"name":"school_education_system_id","type":1,"locator":{"pos":448,"bits":32}},
{"name":"school_umk_id","type":1,"locator":{"pos":480,"bits":32}},
{"name":"school_grade_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"school_subject_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"school_material_type_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"school_exam_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"school_exam_year_id","type":1,"locator":{"pos":512,"bits":32}},
{"name":"category_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"tbk_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"ekn_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"author_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"translator_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"illustrator_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"publisher_series_id","type":1,"locator":{"pos":544,"bits":32}},
{"name":"publisher_id","type":1,"locator":{"pos":576,"bits":32}},
{"name":"publisher_brand_id","type":1,"locator":{"pos":608,"bits":32}},
{"name":"manufacturer_id","type":1,"locator":{"pos":640,"bits":32}},
{"name":"manufacturer_brand_id","type":1,"locator":{"pos":672,"bits":32}},
{"name":"literature_work_cycle_id","type":1,"locator":{"pos":704,"bits":32}},
{"name":"literature_work_cycle_volume_id","type":1,"locator":{"pos":736,"bits":32}},
{"name":"age_restriction","type":6,"locator":{"pos":768,"bits":64}},
{"name":"binding_id","type":1,"locator":{"pos":832,"bits":32}},
{"name":"tag_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"product_set_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"product_type_id","type":1,"locator":{"pos":864,"bits":32}},
{"name":"stationery_format_id","type":1,"locator":{"pos":896,"bits":32}},
{"name":"stationery_line_type_id","type":1,"locator":{"pos":928,"bits":32}},
{"name":"stationery_ink_color_id","type":1,"locator":{"pos":960,"bits":32}},
{"name":"stationery_ink_base_id","type":1,"locator":{"pos":992,"bits":32}},
{"name":"stationery_lead_diameter_id","type":1,"locator":{"pos":1024,"bits":32}},
{"name":"stationery_lead_hardness_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_case_shape_id","type":1,"locator":{"pos":1056,"bits":32}},
{"name":"stationery_mechanism_type_id","type":1,"locator":{"pos":1088,"bits":32}},
{"name":"stationery_diameter_id","type":1,"locator":{"pos":1120,"bits":32}},
{"name":"stationery_feature_id","type":1,"locator":{"pos":1152,"bits":32}},
{"name":"stationery_colors_quantity_id","type":1,"locator":{"pos":1184,"bits":32}},
{"name":"stationery_gender_id","type":1,"locator":{"pos":1216,"bits":32}},
{"name":"stationery_length_id","type":1,"locator":{"pos":1248,"bits":32}},
{"name":"stationery_staple_number_id","type":1,"locator":{"pos":1280,"bits":32}},
{"name":"stationery_stapler_number_id","type":1,"locator":{"pos":1312,"bits":32}},
{"name":"stationery_material_id","type":1,"locator":{"pos":1344,"bits":32}},
{"name":"stationery_punched_sheets_quantity_id","type":1,"locator":{"pos":1376,"bits":32}},
{"name":"stationery_brush_shape_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_brush_material_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_brush_number_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_painting_technique_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_pen_thickness_id","type":1,"locator":{"pos":1408,"bits":32}},
{"name":"stationery_pen_tip_shape_id","type":1,"locator":{"pos":1440,"bits":32}},
{"name":"stationery_mounting_type_id","type":1,"locator":{"pos":1472,"bits":32}},
{"name":"stationery_calculator_capacity_id","type":1,"locator":{"pos":1504,"bits":32}},
{"name":"stationery_calendar_year_id","type":1,"locator":{"pos":1536,"bits":32}},
{"name":"stationery_calendar_type_id","type":1,"locator":{"pos":1568,"bits":32}},
{"name":"stationery_calendar_subject_id","type":1,"locator":{"pos":1600,"bits":32}},
{"name":"stationery_clasp_type_id","type":1,"locator":{"pos":1632,"bits":32}},
{"name":"stationery_compartments_quantity_id","type":1,"locator":{"pos":1664,"bits":32}},
{"name":"stationery_cover_binding_id","type":1,"locator":{"pos":1696,"bits":32}},
{"name":"stationery_cover_surface_id","type":1,"locator":{"pos":1728,"bits":32}},
{"name":"stationery_package_type_id","type":1,"locator":{"pos":1760,"bits":32}},
{"name":"stationery_universal_id","type":1,"locator":{"pos":1792,"bits":32}},
{"name":"stationery_sheets_quantity_id","type":1,"locator":{"pos":1824,"bits":32}},
{"name":"stationery_color_id","type":1,"locator":{"pos":1856,"bits":32}},
{"name":"stationery_volume_id","type":1,"locator":{"pos":1888,"bits":32}},
{"name":"comic_universal_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_character_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_genre_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_series_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_type_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_line_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_section_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_subject_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"game_player_quantity_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"game_audience_id","type":1,"locator":{"pos":1920,"bits":32}},
{"name":"game_child_age_id","type":1,"locator":{"pos":1952,"bits":32}},
{"name":"game_series_id","type":1,"locator":{"pos":1984,"bits":32}},
{"name":"game_use_case_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"game_skill_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"game_duration_id","type":1,"locator":{"pos":2016,"bits":32}},
{"name":"constructor_nation_id","type":1,"locator":{"pos":2048,"bits":32}},
{"name":"constructor_detail_quantity_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"constructor_equipment_type_id","type":1,"locator":{"pos":2080,"bits":32}},
{"name":"souvenir_reason_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"souvenir_format_id","type":1,"locator":{"pos":2112,"bits":32}},
{"name":"souvenir_set_quantity_id","type":1,"locator":{"pos":2144,"bits":32}},
{"name":"toy_type_id","type":1,"locator":{"pos":2176,"bits":32}},
{"name":"toy_height_id","type":1,"locator":{"pos":2208,"bits":32}},
{"name":"gift_hobby_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"gift_for_children_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"gift_new_year_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"gift_section_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"gift_books_on_interest_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"product_id","type":1,"locator":{"pos":2240,"bits":32}},
{"name":"shop_id","type":6,"locator":{"pos":2272,"bits":64}},
{"name":"city_id","type":1,"locator":{"pos":2336,"bits":32}},
{"name":"shop_brand_id","type":1,"locator":{"pos":2368,"bits":32}},
{"name":"price_retail","type":5,"locator":{"pos":2400,"bits":32}},
{"name":"price_discounted_for_guest","type":5,"locator":{"pos":2432,"bits":32}},
{"name":"price_discount_percent_for_guest","type":5,"locator":{"pos":2464,"bits":32}},
{"name":"price_discounted_for_authorized","type":5,"locator":{"pos":2496,"bits":32}},
{"name":"price_discount_percent_for_authorized","type":5,"locator":{"pos":2528,"bits":32}},
{"name":"price_discounted_for_subscriber","type":5,"locator":{"pos":2560,"bits":32}},
{"name":"price_discount_percent_for_subscriber","type":5,"locator":{"pos":2592,"bits":32}},
{"name":"price_discounted_with_loyalty_card","type":5,"locator":{"pos":2624,"bits":32}},
{"name":"price_discount_percent_with_loyalty_card","type":5,"locator":{"pos":2656,"bits":32}},
{"name":"availability_status","type":1,"locator":{"pos":2688,"bits":32}},
{"name":"availability_quantity","type":1,"locator":{"pos":2720,"bits":32}},
{"name":"hall","type":7,"locator":{"pos":0,"bits":0}},
{"name":"rack","type":7,"locator":{"pos":0,"bits":0}},
{"name":"shelf","type":7,"locator":{"pos":0,"bits":0}},
{"name":"in_shop_locations","type":12,"locator":{"pos":0,"bits":0}},
{"name":"updated_at","type":2,"locator":{"pos":2752,"bits":32}},
{"name":"school_purpose_id_list","type":1073741825,"locator":{"pos":0,"bits":0}}
]
},
"index_settings":{
"hit_format":1,
"blob_update_space":131072
},
"tokenizer_settings":{
"type":2,
"case_folding":"non_cjk"
},
"dictionary_settings":{

},
"words_checkpoint":48,
"max_codepoint_length":3,
"chunk_names":[5908,6171,6074,6066,6067,6223,6122,6167,6168,6319,6225,6238,6363,6227,6305,6306,6307,6355,6354,6356,6398,6406,6405],
"soft_ram_limit":134217728,
"index_id":8217252772307796011

tomat · May 28, 2024, 2:23pm

очень странно откуда тогда берется этот 6021 disk chunk - тк в chunk_names он ни фигурирует, на диске не присутствует

попробуйте проверить indextool еще раз или остановить демон и проверить indextool когда файл индекса выгружен из демона

serhio · May 28, 2024, 2:33pm

Как можно его заставить перечитать?
Повторные попытки проверки так же не проходят.
Но с остановленным демоном не проверял, попробую.

tomat · May 28, 2024, 2:44pm

я и не пойму, что indextool читает если вы привели вывод cat /var/lib/manticore/retail_products/retail_products.meta и там нет 6021 disk chunk

serhio · May 28, 2024, 2:59pm

После остановки запустил проверку:

....
check passed, 305.6 sec elapsed
checking disk chunk, extension 6490, 22(23)...
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check passed, 309.1 sec elapsed
check passed, 309.1 sec elapsed

И еще раз повторно, тоже прошла.
Запустил демона и попробовал еще раз проверку и не прошла:

check passed, 334.8 sec elapsed
checking disk chunk, extension 6486, 20(23)...
WARNING: Index header format is not json, will try it as binary...
WARNING: Unable to load header... Error failed to open /var/lib/manticore/retail_products/retail_products.6486.sph: No such file or directory
FAILED, disk chunk /var/lib/manticore/retail_products/retail_products.6486: prealloc failed: failed to open /var/lib/manticore/retail_products/retail_products.6486.sph: No such file or directory
checking disk chunk, extension 6491, 21(23)...
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check FAILED, 1 failures reported, 339.6 sec elapsed
checking disk chunk, extension 6490, 22(23)...
WARNING: Index header format is not json, will try it as binary...
WARNING: Unable to load header... Error failed to open /var/lib/manticore/retail_products/retail_products.6490.sph: No such file or directory
FAILED, disk chunk /var/lib/manticore/retail_products/retail_products.6490: prealloc failed: failed to open /var/lib/manticore/retail_products/retail_products.6490.sph: No such file or directory
check FAILED, 2 failures reported, 339.6 sec elapsed

Вывод ls:

-rw------- 1 manticore manticore 9,9K мая 28 12:29 /var/lib/manticore/retail_products/retail_products.6074.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:55 /var/lib/manticore/retail_products/retail_products.6167.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:57 /var/lib/manticore/retail_products/retail_products.6168.sph
-rw------- 1 manticore manticore 9,9K мая 28 12:59 /var/lib/manticore/retail_products/retail_products.6171.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:15 /var/lib/manticore/retail_products/retail_products.6223.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:18 /var/lib/manticore/retail_products/retail_products.6225.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:23 /var/lib/manticore/retail_products/retail_products.6238.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:41 /var/lib/manticore/retail_products/retail_products.6306.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:43 /var/lib/manticore/retail_products/retail_products.6307.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:46 /var/lib/manticore/retail_products/retail_products.6319.sph
-rw------- 1 manticore manticore 9,9K мая 28 13:57 /var/lib/manticore/retail_products/retail_products.6356.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:00 /var/lib/manticore/retail_products/retail_products.6363.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:11 /var/lib/manticore/retail_products/retail_products.6407.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:13 /var/lib/manticore/retail_products/retail_products.6408.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:15 /var/lib/manticore/retail_products/retail_products.6419.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:24 /var/lib/manticore/retail_products/retail_products.6453.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:25 /var/lib/manticore/retail_products/retail_products.6454.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:28 /var/lib/manticore/retail_products/retail_products.6455.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:30 /var/lib/manticore/retail_products/retail_products.6464.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:35 /var/lib/manticore/retail_products/retail_products.6491.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:35 /var/lib/manticore/retail_products/retail_products.6492.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:54 /var/lib/manticore/retail_products/retail_products.6497.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:56 /var/lib/manticore/retail_products/retail_products.6507.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:57 /var/lib/manticore/retail_products/retail_products.6514.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:58 /var/lib/manticore/retail_products/retail_products.6520.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:58 /var/lib/manticore/retail_products/retail_products.6524.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:59 /var/lib/manticore/retail_products/retail_products.6527.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:59 /var/lib/manticore/retail_products/retail_products.6529.sph
-rw------- 1 manticore manticore 9,9K мая 28 14:59 /var/lib/manticore/retail_products/retail_products.6530.sph

и meta:

{
"meta_created_time_utc":"2024-05-28T15:00:23.421",
"meta_version":21,
"total_documents":26792742,
"total_bytes":0,
"tid":0,
"schema":{
"attributes":[
{"name":"id","type":6,"locator":{"pos":0,"bits":64}},
{"name":"$_blob_locator","engine":1,"type":6,"locator":{"pos":64,"bits":64}},
{"name":"image","type":7,"locator":{"pos":0,"bits":0}},
{"name":"marketing_status_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"literature_work_publishing_year","type":1,"locator":{"pos":128,"bits":32}},
{"name":"preorder_available_at","type":2,"locator":{"pos":160,"bits":32}},
{"name":"released_at","type":2,"locator":{"pos":192,"bits":32}},
{"name":"rating_star","type":1,"locator":{"pos":224,"bits":32}},
{"name":"rating_average","type":5,"locator":{"pos":256,"bits":32}},
{"name":"rating_weight","type":1,"locator":{"pos":288,"bits":32}},
{"name":"review_count","type":1,"locator":{"pos":320,"bits":32}},
{"name":"purchase_stats_day_avg_count","type":1,"locator":{"pos":352,"bits":32}},
{"name":"is_school_prepare","type":4,"locator":{"pos":384,"bits":1}},
{"name":"is_out_of_class_reading","type":4,"locator":{"pos":385,"bits":1}},
{"name":"school_purpose_id","type":1,"locator":{"pos":416,"bits":32}},
{"name":"school_education_system_id","type":1,"locator":{"pos":448,"bits":32}},
{"name":"school_umk_id","type":1,"locator":{"pos":480,"bits":32}},
{"name":"school_grade_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"school_subject_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"school_material_type_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"school_exam_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"school_exam_year_id","type":1,"locator":{"pos":512,"bits":32}},
{"name":"category_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"tbk_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"ekn_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"author_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"translator_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"illustrator_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"publisher_series_id","type":1,"locator":{"pos":544,"bits":32}},
{"name":"publisher_id","type":1,"locator":{"pos":576,"bits":32}},
{"name":"publisher_brand_id","type":1,"locator":{"pos":608,"bits":32}},
{"name":"manufacturer_id","type":1,"locator":{"pos":640,"bits":32}},
{"name":"manufacturer_brand_id","type":1,"locator":{"pos":672,"bits":32}},
{"name":"literature_work_cycle_id","type":1,"locator":{"pos":704,"bits":32}},
{"name":"literature_work_cycle_volume_id","type":1,"locator":{"pos":736,"bits":32}},
{"name":"age_restriction","type":6,"locator":{"pos":768,"bits":64}},
{"name":"binding_id","type":1,"locator":{"pos":832,"bits":32}},
{"name":"tag_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"product_set_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"product_type_id","type":1,"locator":{"pos":864,"bits":32}},
{"name":"stationery_format_id","type":1,"locator":{"pos":896,"bits":32}},
{"name":"stationery_line_type_id","type":1,"locator":{"pos":928,"bits":32}},
{"name":"stationery_ink_color_id","type":1,"locator":{"pos":960,"bits":32}},
{"name":"stationery_ink_base_id","type":1,"locator":{"pos":992,"bits":32}},
{"name":"stationery_lead_diameter_id","type":1,"locator":{"pos":1024,"bits":32}},
{"name":"stationery_lead_hardness_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_case_shape_id","type":1,"locator":{"pos":1056,"bits":32}},
{"name":"stationery_mechanism_type_id","type":1,"locator":{"pos":1088,"bits":32}},
{"name":"stationery_diameter_id","type":1,"locator":{"pos":1120,"bits":32}},
{"name":"stationery_feature_id","type":1,"locator":{"pos":1152,"bits":32}},
{"name":"stationery_colors_quantity_id","type":1,"locator":{"pos":1184,"bits":32}},
{"name":"stationery_gender_id","type":1,"locator":{"pos":1216,"bits":32}},
{"name":"stationery_length_id","type":1,"locator":{"pos":1248,"bits":32}},
{"name":"stationery_staple_number_id","type":1,"locator":{"pos":1280,"bits":32}},
{"name":"stationery_stapler_number_id","type":1,"locator":{"pos":1312,"bits":32}},
{"name":"stationery_material_id","type":1,"locator":{"pos":1344,"bits":32}},
{"name":"stationery_punched_sheets_quantity_id","type":1,"locator":{"pos":1376,"bits":32}},
{"name":"stationery_brush_shape_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_brush_material_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_brush_number_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_painting_technique_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"stationery_pen_thickness_id","type":1,"locator":{"pos":1408,"bits":32}},
{"name":"stationery_pen_tip_shape_id","type":1,"locator":{"pos":1440,"bits":32}},
{"name":"stationery_mounting_type_id","type":1,"locator":{"pos":1472,"bits":32}},
{"name":"stationery_calculator_capacity_id","type":1,"locator":{"pos":1504,"bits":32}},
{"name":"stationery_calendar_year_id","type":1,"locator":{"pos":1536,"bits":32}},
{"name":"stationery_calendar_type_id","type":1,"locator":{"pos":1568,"bits":32}},
{"name":"stationery_calendar_subject_id","type":1,"locator":{"pos":1600,"bits":32}},
{"name":"stationery_clasp_type_id","type":1,"locator":{"pos":1632,"bits":32}},
{"name":"stationery_compartments_quantity_id","type":1,"locator":{"pos":1664,"bits":32}},
{"name":"stationery_cover_binding_id","type":1,"locator":{"pos":1696,"bits":32}},
{"name":"stationery_cover_surface_id","type":1,"locator":{"pos":1728,"bits":32}},
{"name":"stationery_package_type_id","type":1,"locator":{"pos":1760,"bits":32}},
{"name":"stationery_universal_id","type":1,"locator":{"pos":1792,"bits":32}},
{"name":"stationery_sheets_quantity_id","type":1,"locator":{"pos":1824,"bits":32}},
{"name":"stationery_color_id","type":1,"locator":{"pos":1856,"bits":32}},
{"name":"stationery_volume_id","type":1,"locator":{"pos":1888,"bits":32}},
{"name":"comic_universal_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_character_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_genre_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_series_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_type_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_line_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_section_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"comic_subject_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"game_player_quantity_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"game_audience_id","type":1,"locator":{"pos":1920,"bits":32}},
{"name":"game_child_age_id","type":1,"locator":{"pos":1952,"bits":32}},
{"name":"game_series_id","type":1,"locator":{"pos":1984,"bits":32}},
{"name":"game_use_case_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"game_skill_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"game_duration_id","type":1,"locator":{"pos":2016,"bits":32}},
{"name":"constructor_nation_id","type":1,"locator":{"pos":2048,"bits":32}},
{"name":"constructor_detail_quantity_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"constructor_equipment_type_id","type":1,"locator":{"pos":2080,"bits":32}},
{"name":"souvenir_reason_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"souvenir_format_id","type":1,"locator":{"pos":2112,"bits":32}},
{"name":"souvenir_set_quantity_id","type":1,"locator":{"pos":2144,"bits":32}},
{"name":"toy_type_id","type":1,"locator":{"pos":2176,"bits":32}},
{"name":"toy_height_id","type":1,"locator":{"pos":2208,"bits":32}},
{"name":"gift_hobby_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"gift_for_children_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"gift_new_year_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"gift_section_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"gift_books_on_interest_id_list","type":1073741825,"locator":{"pos":0,"bits":0}},
{"name":"product_id","type":1,"locator":{"pos":2240,"bits":32}},
{"name":"shop_id","type":6,"locator":{"pos":2272,"bits":64}},
{"name":"city_id","type":1,"locator":{"pos":2336,"bits":32}},
{"name":"shop_brand_id","type":1,"locator":{"pos":2368,"bits":32}},
{"name":"price_retail","type":5,"locator":{"pos":2400,"bits":32}},
{"name":"price_discounted_for_guest","type":5,"locator":{"pos":2432,"bits":32}},
{"name":"price_discount_percent_for_guest","type":5,"locator":{"pos":2464,"bits":32}},
{"name":"price_discounted_for_authorized","type":5,"locator":{"pos":2496,"bits":32}},
{"name":"price_discount_percent_for_authorized","type":5,"locator":{"pos":2528,"bits":32}},
{"name":"price_discounted_for_subscriber","type":5,"locator":{"pos":2560,"bits":32}},
{"name":"price_discount_percent_for_subscriber","type":5,"locator":{"pos":2592,"bits":32}},
{"name":"price_discounted_with_loyalty_card","type":5,"locator":{"pos":2624,"bits":32}},
{"name":"price_discount_percent_with_loyalty_card","type":5,"locator":{"pos":2656,"bits":32}},
{"name":"availability_status","type":1,"locator":{"pos":2688,"bits":32}},
{"name":"availability_quantity","type":1,"locator":{"pos":2720,"bits":32}},
{"name":"hall","type":7,"locator":{"pos":0,"bits":0}},
{"name":"rack","type":7,"locator":{"pos":0,"bits":0}},
{"name":"shelf","type":7,"locator":{"pos":0,"bits":0}},
{"name":"in_shop_locations","type":12,"locator":{"pos":0,"bits":0}},
{"name":"updated_at","type":2,"locator":{"pos":2752,"bits":32}},
{"name":"school_purpose_id_list","type":1073741825,"locator":{"pos":0,"bits":0}}
]
},
"index_settings":{
"hit_format":1,
"blob_update_space":131072
},
"tokenizer_settings":{
"type":2,
"case_folding":"non_cjk"
},
"dictionary_settings":{

},
"words_checkpoint":48,
"max_codepoint_length":3,
"chunk_names":[6171,6074,6419,6223,6167,6168,6319,6225,6238,6363,6464,6306,6307,6455,6356,6408,6407,6453,6454,6492,6491,6497,6507,6514,6520,6524,6529,6533,6535,6534],
"soft_ram_limit":134217728,
"index_id":8217252772307796011

tomat · May 28, 2024, 3:15pm

наверное у вас во время работы демона происходит оптимизация индекса и indextool проверяет не ту версию индекса которая используется демоном можете попробовать делать FREEZE таблицы на время проверки, только после обязательно сделайте UNFREEZE

serhio · May 28, 2024, 3:41pm

Это нужно будет всегда делать?
Как проверю напишу результат.

serhio · May 30, 2024, 11:52am

Попробовал сделать с FREEZE/UNFREEZE.
Для начала проверил журнал searchd.log:

[Thu May 30 04:48:20.045 2024] [2859936] rt: table retail_products: optimized progressive chunk(s) 14 ( left 12 ) in 4m 47.7s                    
[Thu May 30 05:04:35.929 2024] [2859936] rt: table retail_products: optimized progressive chunk(s) 17 ( left 12 ) in 5m 54s                      
[Thu May 30 05:10:20.439 2024] [2859932] rt: table retail_products: optimized progressive chunk(s) 14 ( left 12 ) in 4m 52.3s                    
[Thu May 30 05:21:18.176 2024] [2859935] rt: table retail_products: optimized progressive chunk(s) 22 ( left 12 ) in 7m 33.7s                    
[Thu May 30 05:35:27.068 2024] [2859934] rt: table retail_products: optimized progressive chunk(s) 20 ( left 12 ) in 6m 49.2s                    
[Thu May 30 05:50:27.119 2024] [2859933] rt: table retail_products: optimized progressive chunk(s) 18 ( left 12 ) in 6m 44.6s                    
[Thu May 30 06:04:26.022 2024] [2859936] rt: table retail_products: optimized progressive chunk(s) 17 ( left 12 ) in 5m 49.4s                    
[Thu May 30 06:20:23.690 2024] [2859936] rt: table retail_products: optimized progressive chunk(s) 19 ( left 12 ) in 6m 44.8s                    
[Thu May 30 06:36:34.152 2024] [2859934] rt: table retail_products: optimized progressive chunk(s) 20 ( left 12 ) in 8m 2.9s                     
[Thu May 30 06:52:10.247 2024] [2859936] rt: table retail_products: optimized progressive chunk(s) 22 ( left 12 ) in 8m 9.8s

Видно, что выполняет optimizing.

Далее выполнил FREEZE retail_products;
И запустил indextool --check retail_products, прошел без ошибок.
Часть логов:

checking disk chunk, extension 17828, 135(137)...
WARNING: secondary library not loaded; secondary index(es) disabled
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check passed, 222.6 sec elapsed
checking disk chunk, extension 17833, 136(137)...
WARNING: secondary library not loaded; secondary index(es) disabled
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking dead row map...
checking doc-id lookup...
check passed, 223.8 sec elapsed
check passed, 223.8 sec elapsed

После выполнил UNFREEZE retail_products;

Вот часть лога, где затронула в работе:

[Thu May 30 11:18:45.704 2024] [2859932] rt: table retail_products: diskchunk 17830(171), segments 2  saved in 2.065639 (2.066044) sec, RAM saved/new 25045262/3694641 ratio 0.755357 (soft limit 101382274, conf limit 134217728)
[Thu May 30 11:28:50.214 2024] [2859932] WARNING: got ident_all
[Thu May 30 11:28:50.564 2024] [2859936] WARNING: rt common merge: table retail_products: failed to merge /var/lib/manticore/retail_products/retail_products.16478 (interrupted because of shutdown)
[Thu May 30 11:28:50.564 2024] [2859936] rt: table retail_products: optimized progressive chunk(s) 1525 ( left 137 ) in 4h 30m
[Thu May 30 11:28:50.694 2024] [2859932] rt: table retail_products: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=117.240 Mb, time delta=604 sec, took=0.129 sec)
[Thu May 30 11:28:57.564 2024] [2859934] rt: table retail_products: diskchunk 17865(138), segments 6  saved in 10.852948 (10.871578) sec, RAM saved/new 123126014/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Thu May 30 11:29:26.577 2024] [2859934] rt: table products: diskchunk 103354(13), segments 27  saved in 7.500577 (7.501896) sec, RAM saved/new 128934433/1420042 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Thu May 30 11:29:41.254 2024] [2859934] rt: table products: optimized progressive chunk(s) 13 ( left 12 ) in 14s 674ms
[Thu May 30 11:35:09.586 2024] [2859935] WARNING: got ident_all
[Thu May 30 11:35:42.695 2024] [2859933] rt: table retail_products: diskchunk 17867(139), segments 1  saved in 2.453579 (2.454015) sec, RAM saved/new 31004605/0 ratio 0.900000 (soft limit 120795955, conf limit 134217728)
[Thu May 30 11:35:45.382 2024] [2859935] rt: table retail_products: diskchunk 17866(140), segments 4  saved in 11.060136 (11.060547) sec, RAM saved/new 132384104/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Thu May 30 11:36:24.592 2024] [2859932] rt: table retail_products: diskchunk 17871(139), segments 1  saved in 2.467480 (2.467937) sec, RAM saved/new 31004605/0 ratio 0.900000 (soft limit 120795955, conf limit 134217728)
[Thu May 30 11:36:28.663 2024] [2859935] rt: table retail_products: diskchunk 17870(140), segments 4  saved in 12.123265 (12.180998) sec, RAM saved/new 133655803/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Thu May 30 11:37:01.984 2024] [2859932] rt: table retail_products: diskchunk 17875(139), segments 1  saved in 3.772637 (3.773063) sec, RAM saved/new 41060478/0 ratio 0.900000 (soft limit 120795955, conf limit 134217728)
[Thu May 30 11:37:04.446 2024] [2859934] rt: table retail_products: diskchunk 17874(139), segments 4  saved in 13.634629 (13.635298) sec, RAM saved/new 149688596/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728

Видно, что тут:

[Thu May 30 11:28:50.564 2024] [2859936] WARNING: rt common merge: table retail_products: failed to merge /var/lib/manticore/retail_products/retail_products.16478 (interrupted because of shutdown)

было завершение работы, согласно документации.
И далее работа была восстановлена.

Похоже такой момент рабочий, но придется всегда явно делать FREEZE/UNFREEZE.
Планируется ли это на стороне indextool с check?

Проверю еще как пройдет теперь с backup.

tomat · May 30, 2024, 12:14pm

indextool \ indexer никак не взаимодействует с демоном
indexer может только сигнал демону отправить, с предложением ротировать индексы

Если для вас это важно - то лучше создать тикет на Github или воспользоватся сервисом Feature engineering

serhio · May 30, 2024, 12:17pm

Понял, спасибо.