Network Data Transfer Memory Leak

Post by **sandervl** » 7. Jan 2009, 12:42

The huge memory drain observed before is due to the file system cache of Windows. The amount of available physical memory reported by e.g. the task manager drops significantly during high guest disk IO.

The system cache value doesn't appear to be related. Use CacheSet to check the current file cache size: http://technet.microsoft.com/en-us/sysi ... 97561.aspx. That tool shows that the cache increases as fast as the available physical memory decreases.

The same behaviour can be observed with 2.0.6. So far I have not seen any evidence of memory leaks. You're welcome to provide proof to the contrary, but that's probably too much to expect given the pleasant tone of your posts.

jaylittle · Post by **jaylittle** » 7. Jan 2009, 15:02

This has nothing to do with agressive caching in Vista. Absolutely nothing. Feel free to explain how closing a 1 gigabyte VirtualBox VM on my machine after running a few hours (and using the network every bit of the time) can free up 2.0 - 2.5 gigs of memory immediately! Or explain how a 500 megabyte VM can suck up the remaining 2.5 gigs of memory on my machine only to suck up all of the physical memory on the host while trying to transfer partimage disk images over the host network connection? VBox stopped the VM due to it's own memory use! Once I closed the VM, +50% of my physical memory returned instantaneously!

There is a memory leak. The denial in this forum is absolutely astounding. Perhaps if you coded the damn application so that it's exorbitant memory use could actually be tracked via the task manager, it would be much much easier to prove it. Frankly you've given me no way to prove it. You ask me to use the task manager and then when I do so, you mention "oh the task manager doesn't actually see the memory we allocate" or some such nonsense. So why ask me to use the damn task manager? What purpose did it serve other than to blow me off because you didn't think I would actually take the time to do it?

You then said, "maybe it's the HOST networking driver". I say, "how can I track that?" I get no answer. I gave it a shot. But frankly given the "fingers in our ears" attitude here combined with the vendor lock in aspect of VDI images, this product's time in my life is done. Attempting to move to VirtualBox was a mistake on my part. It is a mistake that I'm happy to say that I have rectified thru the purchase and transition to VMWare Workstation.

As far as I can tell, 2.1.0 is a version that was better left to what is clearly a non-existent beta test team. It was an absolute downgrade in every way imaginable from my perspective. Moving back to 2.0.6 was an option sure, but when the dev team is in active denial regarding the existence of these bugs that are VERY VERY easy to replicate, there isn't a whole lot to be done.

Here's an easy replicatable scenario for you bums: Create two VMs. The first one (the server) should have no harddisk associated with it. The second one (the client) should have no harddisk image. Boot both VMs off any one of the variety of Linux LiveCDs available (just make sure the one you choose includes netcat). Make sure both VMs are connected to your host interface using the host interface driver. Make sure the host is a 64 bit Vista box.

1) Assign an IP of 10.0.0.1 to the server. Subnet mask 255.255.255.0.

2) Start netcat on the server with the following command:

nc -l -v -v -p 2025 < /dev/urandom

3) Assign an IP of 10.0.0.2 to the client. Subnect mask 255.255.255.0

4) Start netcat on the client with the following command:

nc -v -v 10.0.0.1 2025 > /dev/null

5) Watch the shit hit the fan.

This isn't a hard bug to replicate. Honestly at this point I get the feeling that nobody within the dev team actually has a 64 bit Vista box to test with. Perhaps they are just luddites running XP64 and are in denial regarding the ability of Vista to function well. Who knows. Who cares. Either way I'm done with this insanity.

Good day.

Post by **vbox4me2** » 7. Jan 2009, 15:05

If vista, disable superfetch, defender and diagnostic*, see VB fly without globbing ram.

Post by **sandervl** » 7. Jan 2009, 15:48

I've got four vista 64 machines sitting under my desk and decided to honour your polite request. Can't reproduce it. Two Ubuntu 8.04 32 bits VMs running with host interface networking and netcat as specified. No memory leaks can be observed.

Same is true on a Vista 32 machine. So I'm still not convinced and insults don't add credibility to your case.

jaylittle · Post by **jaylittle** » 7. Jan 2009, 16:03

sandervl wrote:I've got four vista 64 machines sitting under my desk and decided to honour your polite request. Can't reproduce it. Two Ubuntu 8.04 32 bits VMs running with host interface networking and netcat as specified. No memory leaks can be observed.

So how did you attempt to "observe" these leaks? The only way I can see to track the leakage is to watch the Physical Memory Percentage on the Host Task Manager. With this technique it ought to go up pretty quick. You can verify that VBox is responsible by closing the VM and watching all of that memory suddenly returned to the system. Now when I performed a similar test it was between VMWare and VirtualBox - so perhaps you need to have one of the VMs in the test on a seperate host machine as your "Host Network Inteface" driver may have shortcuts that allow it to bypass the host network interface altogether if the source & target machines are both Virtual Box VMs on the same machine.

jaylittle · Post by **jaylittle** » 7. Jan 2009, 16:04

vbox4me2 wrote:If vista, disable superfetch, defender and diagnostic*, see VB fly without globbing ram.

Wrong. Thanks for playing though. Superfetch is disabled on every Vista box I have by default. Defender stays on. These services have nothing to do with VirtualBox eating all of my physical RAM. If they were eating my physical RAM, then that physical RAM wouldn't be suddenly returned to me for use when Virtual Box had it's VM shut down, would it?

Post by **TerryE** » 7. Jan 2009, 16:31

Jay, this is an interesting investigation that is made very difficult because of your combative and rude stance. It really doesn't help. I just wonder what feedback you'd get from the VM developers at MS or VMware if you started flinging around such insults at them.

We did have a similar leakage issue with 2.0.2 IIRC, but that one was cured by an MS hotfix (see www.virtualbox.org/ticket/2242) which Sander probably has applied to his Vista instances (see http://support.microsoft.com/kb/949700) and you might not, BTW.

No one wants to have a leak of this magnitude. However that in itself is something I cannot reconcile: we have many Vista users, yet you seem to be the only one who is hitting the leak? Surely that itself is quite interesting?

Post by **sandervl** » 7. Jan 2009, 16:32

jaylittle wrote:
vbox4me2 wrote:If vista, disable superfetch, defender and diagnostic*, see VB fly without globbing ram.
Wrong. Thanks for playing though. Superfetch is disabled on every Vista box I have by default. Defender stays on. These services have nothing to do with VirtualBox eating all of my physical RAM. If they were eating my physical RAM, then that physical RAM wouldn't be suddenly returned to me for use when Virtual Box had it's VM shut down, would it?

Yes, it could. File cache space allocated for a process is freed when the process exits.

Neither the physical memory percentage in the task manager nor Process Explorer show any evidence of leaks. I've run the server process on 2nd machine and the client in the VM (and vice versa). No difference and there's no shortcut in the host interface code for VMs on the same machine.

Post by **sandervl** » 7. Jan 2009, 16:35

TerryE wrote:We did have a similar leakage issue with 2.0.2 IIRC, but that one was cured by an MS hotfix (see www.virtualbox.org/ticket/2242) which Sander probably has applied to his Vista instances (see http://support.microsoft.com/kb/949700) and you might not, BTW.

Good point, but that was for very specific usage of an API. We've rewritten the code since then.

jaylittle · Post by **jaylittle** » 7. Jan 2009, 17:09

We did have a similar leakage issue with 2.0.2 IIRC, but that one was cured by an MS hotfix (see www.virtualbox.org/ticket/2242) which Sander probably has applied to his Vista instances (see http://support.microsoft.com/kb/949700) and you might not, BTW.

In the hours spent trying to deal with this issue, at some point I requested a link for and installed that particular hotfix on my machine. No Dice.

As for the rest, I've moved on. This will probably be my last post in this thread and this forum as a whole. Perhaps it's my networking drivers. perhap's it's my AV software (AVG). Perhaps it's Windows Defender. Perhaps it's Vista. Perhaps it's the magic "screw you" fairy has decided to sit with me for a bit in regards to VirtualBox.

Either way, I'm done. I had no memory leaks on my system prior to the installation and resulting use of Virtual Box 2.1.0. VMWare doesn't leak memory. Even the craptacular Virtual PC product put out by Microsoft doesn't leak memory. VirtualBox 2.0.6 doesn't leak memory. VirtualBox 2.1.0 is the only product on this machine that is leaking exorbitant amounts of memory when I perform network transfers using either Host networking or NAT.

On top of this, you keep referencing my combative attitude. The reason for that is simple: You keep telling me that I'm not proving the existence of my problem. I have asked how I could go about proving it. I was given a few answers that turned out to be bunk blow off answers (use task manager - what a crock of crap). Not a one of you developers have provided any methodology for tracking down the source of the leak. I don't write device drivers. I'm a network admin and web developer. I don't write low level code. Debugging low level device driver aspects of a Windows system is not my forte. Since you guys developed these drivers, I was hoping to get more insight on what the next step is. That has not happened.

So the position I'm in left me with the following options:

1) Go back to VirtualBox 2.0.6. This option is only a temporary workaround as 2.0.6 had a couple of bugs that got fixed in 2.1.0 that were annoying me. On top of this, as long as the dev team is denying the existence of this bug in 2.1.0, future versions will likely exhibit the same symptoms. Staying with 2.0.6 for an indeterminate amount of time is simply not an option.

2) Live with it. Sorry but having to restart a VM that I use 20 hours a week every couple of hours isn't something I'm willing to tolerate. If I wanted stability like that, I would install Windows 3.11 and reboot every couple of hours just to keep from entering GPF hell.

3) Try to help the team fix it by having them provide instructions to track down the source of the bug. This has not worked out as the dev team seems reluctant/unable to provide information on determining whether or not their low levels drivers are actually the source of the leak.

4) Move to another product that is proven and does not exhibit known issues.

I tried option 3. Perhaps I wasn't the nicest guy about it, but I've made sincere requests for suggestions on how to determine the root cause of the issue. Most of responses seem to dismiss me as an idiot by recommending tactics that have no chance of working or imply that I am making the whole thing up. Not a one of you has stepped up to the plate with a suggestion how to track down the root cause of the memory leakage. Even though I have uninstalled VirtualBox from the machine in question, I still have the VDI images and Machine config for the VM in question. If a credible suggestion was posted here, my curiousity would get the better of me and I would follow through.

However, nobody here has yet to mention anything outside of task manager (which according to a devs own post can't even track the memory legitimately used by VBox much less what it might be leaking). Perf Mon sounds, nice but my knowledge of that tells me that it isn't going to do anything more than task manager. If you can, feel free to provide specific instructions regarding the settings and counters that should be used to acquire such information.

One way or another, you guys haven't left me with a choice. Regardless of whether or not the bug is fixed, I won't be using Virtual Box as I intend to make good on my 94.50 USD investment into VMWare Workstation. However I wouldn't mind at least providing a helping hand when it comes to fixing a bug that will likely diminish the experience that other/future users have with Virtual Box.

Post by **sandervl** » 7. Jan 2009, 17:26

We have asked you to open a ticket and when doing so you are asked to include your VBox.log as well. The last file would help prevent me asking you 100 questions.

Such reports need detailed information about your guest and host. You can't expect us to look into a magic ball and guess.

I have ignored all your insults and tried to reproduce the problem to no avail. If you don't want to invest any more time and use VMWare, that's fine, but don't claim that we didn't take you seriously. Because if I thought you were an idiot reporting nonsense, I wouldn't have bothered to reply in the first place.

jaylittle · Post by **jaylittle** » 7. Jan 2009, 18:11

Here is the log that was created when I tried to use partimage to transfer an image of one of the partitions over the host network to another vm setup on VMWare:

Link to Log

There was a single Host Interface configured against my wireless card in this configuration. The VM had 1 gig of RAM. This VM eventually ate up all of my host memory and VBox forced it to suspend once it detected a low memory condition on the host.