Memory consumption sql_field_string


#1

Hi together

I have only one question: why my memory consumption on the server is only 8.5 MB with an .sps file with 215 MB? Docs say, .sps would be load in memory…

ver: 2.7.0 GA / Windows Server 2016

best regards
Peter

my config:

source srcWissen
{
type = mysql
sql_db = *
sql_port = *
sql_query = SELECT SQL_NO_CACHE id,Begriff,wKey,T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,tWissenTempl.Text AS Text,tWissenTempl.posOben AS posOben FROM tWissen,tWissenTempl WHERE tWissenTempl.Template = tWissen.Template
sql_host = localhost
sql_user = *
sql_pass = *
sql_field_string = Begriff
sql_field_string = wKey
sql_field_string = Text
sql_field_string = T1
sql_field_string = T2
sql_field_string = T3
sql_field_string = T4
sql_field_string = T5
sql_field_string = T6
sql_field_string = T7
sql_field_string = T8
sql_field_string = T9
sql_field_string = T10
sql_field_string = T11
sql_field_string = T12
sql_field_string = T13
sql_field_string = T14
sql_attr_uint = posOben
sql_query_pre = SET NAMES utf8
}

index tWissen
{
source = srcWissen
path = d:/sphIndex/sphWissen
charset_table = 0…9, A…Z->a…z, _, a…z, U+A8->U+B8, U+B8, U+C0…U+DF->U+E0…U+FF, U+E0…U+FF, U+40
ignore_chars = U+0060
min_word_len = 2
min_prefix_len = 0
min_infix_len = 0
preopen = 1
#mlock = 1
}

searchd
{
listen = 3360:mysql41
log = d:/sphIndex/Log/sphWissen.log
read_timeout = 5
pid_file = d:/sphIndex/Log/sphWissen.pid
workers = thread_pool
net_workers = 1
mysql_version_string = 5.1.73
binlog_path =
seamless_rotate = 1
qcache_max_bytes = 256000000
qcache_thresh_msec = 10
qcache_ttl_sec = 600
}

indexer
{
mem_limit = 64M
write_buffer = 16M
}


#2

Hi @Greyhound. Does mlock = 1 help?


#3

maybe… but it does not work on windows server… still not…
regard sPeter


We just reproduced this issue and I created ticket at Github github#125 You might subscribe there to be informed on progress of fixing this issue.
regards Stan


#4

OK
Answering your question:

the point is that:

  • attributes are mmaped into memory. mmap is a way of working with files when they’re mapped into memory which allows to then work with them in memory and let OS read and write it to disk when needed.
  • OS decides itself whether the file (in your case .sps) at any moment is fully or partly in memory since other processes may need the memory too. That’s why the memory consumption may be lower then .spa + .sps + .spm + .spi
  • mlock (support available in *nix) allows to make sure the attributes are always in memory

So if you don’t suffer from higher query response time due to mlock=0 then just take the above into account. Otherwise I would suggest to migrate to Linux. Let me also remind that the product has commercial support, consulting and development https://manticoresearch.com/professional-support/ and if it’s critical for your business feel free to contact us and we’ll think of a better solution together.


#5

Please tell me, why manticore not works with mlock on windows… all other software has no problem with this, also sphinx 2-3 has no problems.
regards Peter


#6

Which version exactly?


#7

all 2.xx sphinx versions use the full memory for .sps and also accept mlock.
regards Peter


#8

now tested also on sphinx 3.1 - memory consumption full on start and still the same after 12 hours without any query. so mlock works perfect.

regards Peter


#9

I see, thanks for letting us know. I’ve updated https://github.com/manticoresoftware/manticoresearch/issues/125


#10

just for info:

8 Mio records with 4x sql_field_string
average query time:
manticore 2.8.1: 45ms (memory used: 12 mb)
sphinx 3.1.1: 15ms (memory used: 223 mb - and with mlock locked)

You see, the right one memory management is very important for me…
regards Peter


#11

Thanks for letting us know. The point is that very little %% of users run Manticore (or Sphinx) under Windows Server in production. So it’s just a matter of goals and priorities. We will of course prioritize looking into this issue in future, but unfortunately right now we cannot afford working on that since we have other priorities: issues from our support clients, other features that most users need (e.g. replication and Manticore v3). Don’t get me wrong, I’m not saying Windows users and their issues are not important for us. It’s just that we won’t be able to run the business successfully if we do less priority things first. We’re a small company and have to choose how we spend the resources.
As I already said for those whose business suffers from any issues we have a commercial support. We’re always working on our clients’ issues and feature requests first.