Server Stops Processing Data After Importing 20,000 Out of 60 Million Products

General Discussions - English
1

Hello everyone,

I’m encountering an issue while writing data to my server. I am currently importing 60 million products, but after importing approximately 20,000 products, the server often stops processing additional data and returns the following error message:

2024-06-28T18:36:00.159+0200    INFO    repository/mtc_product_repository.go:113        Failed to execute bulk indexing {"retryCount": 0, "error": "Post \"http://localhost:9308/bulk\": read tcp 127.0.0.1:50890->127.0.0.1:9308: read: connection reset by peer"}
2024-06-28T18:36:00.159+0200    INFO    repository/mtc_product_repository.go:113        Failed to execute bulk indexing {"retryCount": 0, "error": "Post \"http://localhost:9308/bulk\": read tcp 127.0.0.1:50874->127.0.0.1:9308: read: connection reset by peer"}

I am currently using a Docker container for this process, and I suspect that the issue might be related to resource exhaustion or some other limitation within the container.

Does anyone have an idea what might be causing this issue and how I can resolve it?

Thank you in advance for your help!

2

What do you mean by “it stops processing”? Does it hang or just reject another batch?

3

By “it stops processing,” I mean the server rejects new batches and returns an error.

It seems the connection is reset by the peer.

4

What error does it return and what’s in docker logs? read: connection reset by peer seems to be an error from your app.

5

I addressed this by implementing retry logic with exponential backoff and reducing the bulk size. but is not a nice solution

6

When I execute the HTTP request in my code, the above error occurs:
The error in your code occurs when the HTTP request (req) to the HTTP client fails. This happens within the call

	resp, err := repo.apiClient.GetConfig().HTTPClient.Do(req)
		if err != nil {
			repo.provider.logger.Info("Failed to execute bulk indexing", zap.Int("retryCount", retryCount), zap.Error(err))

docker logs do not show any errors.

on the second or third attempt, it will take over the data.

The message “connection reset by peer” means that the connection was unexpectedly terminated by the other side (the “peer”).

7

Does increasing max_packet_size to 128m help?

8

After adding this value and restarting Docker, MySQL becomes unresponsive:

My configuration:

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9308:http
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
    max_packet_size = 128M
}

SQL output:

root@68a3e272eea1:/var/lib/manticore# mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

without max_packet_size it works.

9

hey @Sergey, i have solved the problem by reducing the batch.

Now it works well and fast.

Still the question remains why I can’t add the value “max_packet_size” to the “searchd”.

10

Because just mysql attempts connecting via /var/run/mysqld/mysqld.sock which you are not listening on.

Instead of mapping the config, you can do -e searchd_max_packet_size=128m:

snikolaev@dev2:~$ docker run -e EXTRA=1 --name manticore --rm -d -e searchd_max_packet_size=128m manticoresearch/manticore:6.3.2
6631bd61af23444d7c540cbbd0c63cfe8c1bdbc6caf318103e708f85d81992db
snikolaev@dev2:~$ docker exec -it manticore mysql
mysql> show settings;
+-------------------------+----------------------------------------+
| Setting_name            | Value                                  |
+-------------------------+----------------------------------------+
| configuration_file      | /etc/manticoresearch/manticore.conf.sh |
| worker_pid              | 1                                      |
| searchd.listen          | 9306:mysql41                           |
| searchd.listen          | /var/run/mysqld/mysqld.sock:mysql41    |
| searchd.listen          | 9308:http                              |
| searchd.listen          | 172.17.0.8:9312                        |
| searchd.listen          | 172.17.0.8:9315-9325:replication       |
| searchd.max_packet_size | 128m                                   |
11

yes. that’s a good idea. thanks for the hint