Large datasets setup

General Discussions - English
1

Hello,

We currently have a distributed index with about 100 plain indexes spread out over 4 nodes.
The indexes are between 20GB and 90GB (all index files combined) each.

We have been having some issues lately with system memory and were wondering what the most efficient setup would be:

  • a small number of very large indexes on few large nodes
  • a large number of smaller indexes (20GB) on more ‘lighter’ nodes

What would be considered an optimal size for a plain index?

Thanks in advance

2

In theory, fewer shards may be beneficial as the full-text dictionaries would be duplicated fewer times. In practice, RAM overconsumption is often caused by the attributes. It’s worth considering whether switching to columnar storage can help, or if reviewing and optimizing the schema is a better approach.

3

Thank you for the quick reply, I took a look at the columnar storage but it’s not completely clear what’s meant when saying ‘when the index is larger than RAM’: 1 index size or all indexes on the node combined size?

Also, what kind of performance decrease could we expect when switching to columnar storage?

4

1 index size or all indexes on the node combined size?

.spa, .spb and .spi files of the tables that are used in searches.

Also, what kind of performance decrease could we expect when switching to columnar storage?

Please review the benchmarks on https://db-benchmarks.com/. For example:

image
image1920×2390 253 KB