On defragmentation and NTFS compression (Windows host and guest)

This is for discussing general topics about how to use VirtualBox.
Post Reply
LMF5000
Posts: 4
Joined: 23. Apr 2017, 20:50

On defragmentation and NTFS compression (Windows host and guest)

Post by LMF5000 »

I have two questions regarding best practices with virtual disk image files.

Assume both guest and host windows, and every file system is NTFS. I want to maximise the performance of the guest. The .vdi file is fixed-size, and on an NTFS partition on a dedicated mechanical drive. My questions are:

1. Usually NTFS file system compression speeds up IO operations (I have a powerful CPU and slow hard disk so I gain more speed from reducing the read/write volume than I lose due to CPU overhead). For best performance, which should be compressed, the host filesystem (volume on which the .vdi file is), or the "C" drive as seen by the guest?
2. Which should be defragmented, the .vdi file (as seen by the host), or the "C" drive (as seen by the guest)?

Right now I have a Win 10 guest running on a Win 7 host. I've enabled NTFS compression at both levels (the physical volume is compressed and the Windows 10's primary partition is compressed too).

For question #2 I need some help. Running a defrag utility within windows 10 creates a nicely defragged filesystem (as seen by the guest). But looking at the resulting .vdi file externally from the host, it appears fragmented.
If I defragment the .vdi using the host, it then creates a fragmented hard disk as seen by the guest. So which should be defragmented for maximum performance - the host or guest filesystem?
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by mpack »

LMF5000 wrote:I want to maximise the performance of the guest. The .vdi file is fixed-size
Well, bad move right there in that case. What made you think that the VDI file being a certain size makes it go faster? There is no logical reason to expect this to make any difference whatever except that people tend to make fixed sized disks too small, creating congestion in the VM.

Most of the rest of your post proceeds from similar axioms that I would struggle to accept without evidence, e.g. binary data typically isn't very compressible by lossless methods so I find it hard to believe that compressing the host drive buys you much there. OTOH seek times in large compressed files are going to be horrible I'd have thought, access times to very small files would be bad as well as they compress even less.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by socratis »

I'll focus on another part; disk compression. Without going into a lot of details on why it's a bad idea (that's why it's not the default option), 1) disk compression is NOT recommended for the system files (for a good reason) and 2) it will only benefit if the compressed files can fit effectively into memory. Not sure about your setup, but if you can fill your whole compressed VDI (tens of GB) in your RAM, I believe you're looking at the wrong solution vector.

I believe you're over-complicating the problem and you're trying to squeeze "performance" from somewhere that you cannot get any performance improvements. Just run some tests with and without compression and compare the results.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Petr Vones
Posts: 89
Joined: 27. Dec 2012, 01:20
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows 10 64-bit
Location: Czech Republic

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by Petr Vones »

LMF5000 wrote:on a dedicated mechanical drive
Is your issue. SSD performance is noticeable there. Plus you don't have to care about fragmentation.

Also check Host I/O cache settings https://www.virtualbox.org/manual/ch05.html#iocaching
LMF5000
Posts: 4
Joined: 23. Apr 2017, 20:50

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by LMF5000 »

mpack wrote:
LMF5000 wrote:I want to maximise the performance of the guest. The .vdi file is fixed-size
Well, bad move right there in that case. What made you think that the VDI file being a certain size makes it go faster?
When I installed Win10 to a dynamically-expanding .vdi, writes were very slow, and scattered all over the host partition in multiple file fragments as it "added-on" to the file. You have to watch a fresh Win 10 install trying to update itself to really appreciate how much disk access it does. Under this setup, I was seeing a throughput of only around 1.5 to 5MB/s. By defragmenting, cloning to a fixed-size .vdi, and defragmenting that file on the host, my throughput rose to 3 - 10 MB/s, which is still slow, but twice as fast. The raw disk has a top speed of about 60MB/s (it's a 160GB 2.5" Samsung found in an old laptop, made circa 2007).

I don't think anyone really addressed my questions directly - I take it compression should be kept OFF in host and guest? What about defragmenting - host, guest or both?
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by mpack »

Fixed and Dynamic VDIs use an identical data structure, so you can't possibly have observed a difference due to the VDI data structure. The only difference is whether the blocks are allocated on VDI creation or on demand. Fragmentation is possible with either, depending on the state of the host filesystem - and in any case you already said you defragged the host, so that can't make a difference either.
LMF5000
Posts: 4
Joined: 23. Apr 2017, 20:50

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by LMF5000 »

mpack wrote:Fixed and Dynamic VDIs use an identical data structure, so you can't possibly have observed a difference due to the VDI data structure. The only difference is whether the blocks are alllocated on creation or on demand. Fragmentation is possible with either, depending on the state of the host filesystem - and in any case you already said you defragged the host, so that can't make a difference either.
Of course, the difference I observed was because the dynamically-expanding vdi was allocating space on-the-fly as windows 10 was writing (and this on a highly used data partition where much fragmentation resulted), whereas the fixed-size vdi already allocated all 35GB so the writes were sped up (by a factor of 2).
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by mpack »

That's a short term effect: there are a finite number of blocks and each is only ever allocated once, whether you do it upfront (fixed) or on demand (dynamic). But of course in the latter case there's no overhead for blocks you don't allocate. Hopefully you didn't base a long term strategy on this short term effect.
LMF5000
Posts: 4
Joined: 23. Apr 2017, 20:50

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by LMF5000 »

mpack wrote: Hopefully you didn't base a long term strategy on this short term effect.
I choose the most suitable one depending on the case. For most machines I use dynamically expanding, but for this long-term win10 guest a fixed-size performed much better in my opinion.

So, what are your thoughts on defragmentation and NTFS compression?
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by mpack »

I addressed the latter in my first post, at least as much as I want to.
ams_tschoening
Posts: 24
Joined: 24. May 2017, 16:30

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by ams_tschoening »

LMF5000 wrote:So, what are your thoughts on defragmentation and NTFS compression?
As already have been said, buy a SSD. Neither defragmentation nor compression will provide you the performance you want on a mechanical drive. Both might improve your performance, but you already tested things and recognized that defragmenting the guest fragments the host and the other way around. In theory, compressing the VDI image and the guest shouldn't harm at all, but I really doubt it provides the performance gain your are hoping to see. Additionally, take the time into account you need to compress things, defragment the guest from time to time and think of other things influencing your performance: The host Windows will create things like snapshots in case of Windows updates, which essentially copy things around after a snapshots has been created and blocks are changed. Those things put additional I/O load on your machine, especially if you have a snapshots created by Windows and afterwards defrag your VDI and rewrite it at large.

It's not worth it, compression and defragmentation won't get you magnitudes of better performance. If your HDD is the bottleneck, replace it with some SSD, your possibilities to optimize that bottleneck in software are very limited.
ams_tschoening
Posts: 24
Joined: 24. May 2017, 16:30

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by ams_tschoening »

mpack wrote:That's a short term effect: there are a finite number of blocks and each is only ever allocated once, whether you do it upfront (fixed) or on demand (dynamic).
You are completely missing the point: In case of reserving storage for a fixed size image, every part of the software stack can work optimized: VirtualBox tells the OS that it needs X GB of storage, the OS tells that to the file system and that aquires the needed storage as optimized as possible, meaning in large, linear chunks of data. This is not even possible in case of dynamic images, so the storage needed might be cluttered all over the drive as needed, 4 KB of storage here, 1 MB in linear there, 256 KB over there and so one. And without things like defragmentation that won't ever change anymore, so the storage for the image is cluttered all over the drive and the guest now wants to store e.g. images with 100 MB of data. The guest most likely writes that data linear to it's underlying storage, without knowing it's cluttered all over the underlying physical disk and VirtualBox has no other chance than to clutter the needed 100 MB of data into the sectors it dynamically allocated ten years before, with no linear block of storage able to take the 100 MB of data... So yes, fixed size disks might make a huge difference and therefore are recommended in EVERY setup dealing with old school, not random-access storage like HDDs, if performance is critical.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by socratis »

ams_tschoening wrote:So yes, fixed size disks might make a huge difference and therefore are recommended in EVERY setup dealing with old school, not random-access storage like HDDs, if performance is critical.
You might want your suggestion to companies that deal with databases. They'll appreciated it. Start with Google, they use a lot of these old school HDDs. And they're paramount about their performance, as far as I know.

I think you guys are making a mountain out of a molehill regarding fragmentation. Seriously now, it's been years and years since anyone talked about defragmentation and performance in the same sentence. This is not 1995 anymore.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: On defragmentation and NTFS compression (Windows host and guest)

Post by mpack »

ams_tschoening wrote: You are completely missing the point: ...
I doubt it. You argued in one post that defragmentation buys marginal benefits at best, and then in the very next post (aimed at me) you argue that the downsides of fixed size drives are acceptable because of the same marginal performance benefit. I find your views inconsistent, and your understanding of my position flawed.

My position is that people are frightened by the size of fixed size disks. Therefore they make them too small. Therefore they suffer from congestion inside the guest OS. I don't think I can have been much clearer on this point.

And if your recommendation is to use SSDs then the entire discussion is moot: fragmentation is a meaningless concept when seek time is 0. The marginal benefit is reduced to 0 - only the downsides remain.
Post Reply