in index status, does ram_bytes include ram_chunk?

General Discussions - English
1

As subject really, in show index ... status (for RT index!) does the ram_bytes figure include the ram_chunk or not?

ie when trying to memory estimate usage, should be using ram_bytes+disk_mapped_cached+ram_chunk
or just
ram_bytes+disk_mapped_cached

Hopefully a easy question, I just can’t tell for sure.

2

for plain \ disk chunks of the RT index it calc as

pRes->m_iRamUse = pRes->m_iMappedResident;
...
pRes->m_iMappedResident = m_tAttr.GetCoreSize ()
			+m_tBlobAttrs.GetCoreSize ()
			+m_tWordlist.m_tBuf.GetCoreSize ()
			+m_tDeadRowMap.GetCoreSize ()
			+m_tSkiplists.GetCoreSize ();

ie includes amount of data attr \ blobs \ dictionary \ skiplist actually mapped into app memory

for the RT index itself it includes all disk chunks ram_bytes plus the RT RAM itself - the amount of data stored in the RAM segments (shown as ram_chunk)

pRes->m_iRamUse = sizeof( RtIndex_c ) + pRes->m_iRamChunkSize;

int64_t iUsedRam = SegmentsGetUsedRam ( tGuard.m_dRamSegs );
pRes->m_iRamChunkSize = iUsedRam + tGuard.m_dRamSegs.GetLength()*int(sizeof(RtSegment_t));

ie

  • ram_bytes is the total amount of RAM used by the RT index
  • ram_chunk the RAM used by the only RT part of the RT index (should be around zero after RAM segments flushed into disk chunk)
3

Ok. Its just that I have

                   indexed_documents 29,975
                       indexed_bytes 2,998,480
                           ram_bytes 94,233,040
                          disk_bytes 345,103,916
                         disk_mapped 166,225,388
                  disk_mapped_cached 45,056
                    killed_documents 30,151
                           ram_chunk 94,168,080
            ram_chunk_segments_count 24
                         disk_chunks 4

94Mb seems beleivable for hte RAM Chunk, but with 4 disk chunks, dont see how 64k is all used by disk chunks (the spa files are bigger than that)

94,233,040-94,168,080 = 64960

4

maybe OS does not load them into memory but keep them to disk?
if you use mlock option and it succeed only then you could be sure the data actually in memory

5

Ah, now got

                           ram_bytes 43,040,592
                          disk_bytes 517,045,790
                         disk_mapped 249,016,738
                  disk_mapped_cached 43,012,096
                           ram_chunk 0
                         disk_chunks 6

Its obvisoully, flushed the RAM chunk in the meantime, but did some more queries (with MATCH and attribute) and have much higher disk_mapped_cached now

… would explain the 64k diff as disk_mapped_cached was only 45k.

I’ve always assumed attributes and dictionary etc are always mapped into memory (even if unused), it was only the extra files (like doc lists) that loaded as needed. Seems was wrong about that (although yes, reading about mlock, can influence it!)

So really ram_bytes = disk_mapped_cached + ram_chunk + (bit extra for headers etc)

“peak” (worst case) memory usage woudl be basically
= disk_mapped + mem_limit + (bit extra for headers etc)

(ie total of disk_mapped COULD be mapped to memory, even if not currently. And mem_limit should be the biggest ram_chunk can get to)

Thanks for helping me understand it!