Manticore as a load balancer.

General Discussions - English
1

hey !
I am currently handling a large volume of log data generated every second, which needs to be inserted into a Manticore table/index called logs. Inserting all these records into a single Manticore server could potentially cause a crash due to the high load. To mitigate this, I have employed Nginx as a load balancer to distribute the insertion workload across multiple Manticore servers, such as serverA_manticore, serverB_manticore, serverC_manticore, and so on.

I am seeking an alternative approach to load balancing or a native solution within Manticore that supports this kind of distributed insertion behavior. Thank you.

2

why not use Logstash with distributed pipeline?

3

I have configured Fluent Bit to monitor the log file and utilize its output plugin to seamlessly insert the log data into Manticore. The configuration file fluent-bit.conf specifies the HTTP output plugin, directing the data flow to the designated load balancer, identified by ${LOADBALANCER_HOST} and ${LOADBALANCER_PORT}.

fluent-bit.conf:
[OUTPUT]
Name http
Match *
Host ${LOADBALANCER_HOST}
Port ${LOADBALANCER_PORT}
etc …

4

This is planned and there’s some progress, but it’s far from completeness yet - Manticore public roadmap (large tasks only planned for the nearest future, not bugfixes / smaller issues) · GitHub

1 Like