5.2.10 - consuming more RAM than expected.

[NiteRain]

System specs are as follows:
.

• 62.8GB of RAM
  1TB of Hard drive space - Samsung 960 Pro M.2
  CPU is an AMD Threadripper 1950x - 16Core
  Running Ubuntu 18.04 LTS

So I started small, I want to start with running 5 VMs at the same time, so I set them all to have 2 processors, and use 2048MB of RAM. What I am seeing is my headless VMs consuming more than 7 times the memory expected (RES). I have attached a screenshot of one of the headless processes running, if I start all of them up, they will consume around the same amount of space, or more, and then use up the swap space.

On a smaller system using vmware workstation, I seem to stay within the allocated memory space, is there a setting I am missing?

[socratis]

is there a setting I am missing?

Cache, the host's cache.

[NiteRain]

Hi Socratis, are you talking about the Host IO Cache setting? If you are, that is currently set to on.

[socratis]

No, I was talking about your host's file caching. Are you using shared folders and accessing files on your host?

[Martin]

The files containing the virtual harddisks are also cached on the host.

[NiteRain]

No, I was talking about your host's file caching. Are you using shared folders and accessing files on your host?

0.3 used shared folders for cuckoo, so it is very likely this version also uses shared folders as well. However, I don't think this would increase the memory of each VM to 11G. We are only sharing one file from 0.1 to 10MB each time. And we are always reverting back to an initial-state as well.

I did come across a thread I found interesting, which talks about allocation (viewtopic.php?f=7&t=86889), and it seems I shouldn't be having this issue -- unfortunately the person who started the thread ended up going to another vm. I have cut down to 3 vms; I am still getting close to consuming all of my RAM, but these vms do not run for very long,

[socratis]

I have cut down to 3 vms; I am still getting close to consuming all of my RAM

I'm using no VMs at the time of this writing. I'm almost at 80% of my RAM consumption, and if I keep on browsing I'll hit the max pretty soon. It's called "caching".

0.3 used shared folders for cuckoo

I don't even know that that may mean in the context of the current discussion.

However, I don't think this would increase the memory of each VM to 11G. We are only sharing one file from 0.1 to 10MB each time.

That's not how caching works. You cache not when the file is open, you cache things for when the file is not open so that you can retrieve it faster.

And we are always reverting back to an initial-state as well.

Nothing to do with your question at hand.

Something's not right in the way that the information is interpreted. For example, the "%" in your "top" output, doesn't make sense. You can't have 6 processes * 21.8 % of the RAM. That's 131% of something, doesn't compute.

[NiteRain]

I think I now understand where we are missing one another.

My display is htop, not top. Htop displays caching as yellow in the memory bar, however, 14.6G/62.8G is what is being used, cache isn't part of that number, for example https://i.stack.imgur.com/Sw4l6.png this is htop where cache is taking up all the remaining RAM, notice 1901/31866MB; I believe in top it would probably show 31866MB/31866MB since it calculates that using caching. Also the 21.8% of RAM and all of the processes with the same numbers, are threads of the same process, another thing htop is able to do is give you a tree view, so you can see which processes/threads are linked, I probably should have used a tree view to drive my issue home, so all of those processes together are only taking up 21.8%, I was only running one at the time. Imagine if I were running 5 or 10.

The cuckoo reference was my attempt to answer your shared folders question, an older version of the application that I am running used shared folders, I didn't want to look through the source code to confirm if the one I am using now did it differently.

We would really appreciate any help on this matter.