Downsizing a Windows XP installed partition

[fmillion]

Hello,

I am having problems with downsizing a Windows XP partition.

First off, here's what I want to do. I have installed a Windows XP machine. I then made a clone of the machine and removed a bunch of software installed in the initial machine from the clone. This freed up considerable disk space, so I wanted to shrink the disk image of the clone.

I'm on an SSD based machine so I like to use the "fixed size" allocation. To me, this mens that the SSD "blocks" will be designated for the disk image, and performance will not degrade due to the VDI becoming fragmented across the large SSD blocks.

I have actually successfully followed this procedure before, but this time it's not working.

Here's how I'm producing the clone:

1. Creating a new VDI image, of the desired (smaller) size, and attaching it to the IDE controller as primary master.
2. Booting Linux GRML from an ISO image.
3. Partitioning the new VDI image (it'd be /dev/sdb in Linux) using fdisk.
4. Using ntfsresize to size down the NTFS filesystem on the original (larger) disk to a size that will successfully clone to the smaller disk.
5. Using ntfsclone to clone the NTFS filesystem from the bigger disk to the smaller disk.
6. Using ntfsresize again to ensure the new filesystem on the smaller disk fills the entire partition.
7. Copying the MBR from the original disk to the new disk - only copying the first 446 bytes so as to not overwrite the new partition table. (dd if=/dev/sda of=/dev/sdb bs=446 count=1)
8. Shutdown, detach old disk, assign new disk to be primary master, attempt to boot.

Doing this the first time yielded a frozen non-blinking cursor.

So I booted XP's recovery console and executed the fixboot and fixmbr commands. This did not fix the problem.

I had just done the above steps using the latest (2011.12) release of GRML. Upon closer inspection, it looks like the fdisk tool included in the newest version is altering the CHS values of disks so as to ensure that the first partition always starts on sector 2048 rather than sector 63. This would actually be desirable for an SSD, but I decided it'd be a matter for another day. So I booted into an older release (2011.05) which I have actually used before to do this very procedure, and repeated all the above steps.

This time I got the message: "A disk read error has occurred press Ctrl+Alt+Del to reboot."

Thinking this might have something to do with CHS being screwy, I zeroed out the disk in GRML then booted the XP install CD and partitioned and formatted the disk from within there. I figured this means the only thing I'd be adding back into the mix is my NTFS filesystem. I then again attached the old disk and cloned the partition, this time not touching the MBR at all. The "disk read error" message persists.

I'm not sure where to go from here. I have a hunch this has to do with CHS mismatches. I did force fdisk on GRML to use 255h/63s, which is the same geometry the old larger disk image used - in this case only the cylinder count differed. This didn't help at all either - same message.

Since I have done this very procedure before to downsize an installed XP, something has to be wrong somewhere, either with this particular XP installation or with the latest VBoxes (I think I last did this on 4.1.2) or with my procedure itself.

Does anyone have any advice?

Thank you,

F

[vbox4me2]

CloneVDI tool to shrink the VDI, create a second fixed size VDI and use clonezilla to copy over the shrinked contents to the new fixed vdi disk. Done.

[mpack]

CloneVDI tool to shrink the VDI, create a second fixed size VDI and use clonezilla to copy over the shrinked contents to the new fixed vdi disk. Done.

I'm afraid that will do nothing. CloneVDI will eliminate unused blocks from a dynamic host image, yes. However using CloneZilla to copy the image to a fixed size drive will simply add the unused blocks back again. It just makes no sense to talk about compaction in the context of a fixed size image.

The only way to shrink a fixed size drive is with gparted (or the tools which gparted is a front end for, such as ntfsresize). This removes partition capacity, it is not compaction.

@fmillion. I don't really get why you think a fixed size drive gives you any advantage, but each to his own I guess. In particular your talk of fragmentation in an SSD (which has no significant seek latency) doesn't make much sense to me.

[vbox4me2]

Ok forgot gparted but thats all.

CloneVDI tool to shrink the VDI, run gparted liveCD to downsize the partition, create a second fixed size VDI and use clonezilla(or gparted) to copy over the shrinked and downsized contents to the new fixed vdi disk. Done.

[fmillion]

@vbox4me2: I'll try the cloneVDI method and post back. One plus-side of using the ntfsresize/ntfsclone tools is you don't literally have to resize the source disk's partition, because the tools depend on the NTFS itself for sizing information. From what I'm seeing CloneVDI works at a lower level and thus the partition will need to be resized to fit the smaller disk as well as the filesystem itself. Also, since I'm on OSX I'll have to find a Mac version.

I still am not quite sure why the ntfsclone method failed this time around. The status display for VBox shows that 9KB of data was read off the hard drive device - this corresponds to the 512 byte boot sector, the 8KB of NTFS load code and some other 512 byte block (maybe the last filesystem block). So it looks like everything's being read in properly. My knowledge of BIOS based booting however tells me that this is one of the most critical stages (and one of the most delicate) and it can be thrown completely out of whack by the simplest details.

@mpack: Here's my rationale for using fixed-size. You probably know that all SSDs do erase/write in blocks that are far larger than the normal filesystem cluster size. For example, let's assume a particular SSD's smallest possible erase block size is 128KB, but NTFS's cluster size is 4KB.

Now, if I use a dynamically allocating disk, the disk expands as I use sectors on it. The problem is these sectors will never be anywhere near aligned with the host filesystem's blocks. So say that I write 4KB to my virtual disk, and it expands on the SSD. Now, to write that 4KB, the SSD has to erase then write 128KB.

Now, I use my host system a bit, and something I do occupies the rest of that 128KB. Now, I use the VM again and occupy another few KB. And so on and so on.

Quickly you can see you end up in a situation where even a few contiguous sectors in the VM could be spread across many physical blocks on the SSD. In extreme cases you could even see a situation where writing 1MB to the VM image would need hundreds of erase/write cycles on the underlying SSD!

You're right that fragmented files don't impact read performance on an SSD, but they can (and do) impact write performance. Sure, modern SSD firmware can try to mitigate this a bit, but until we see SSDs that are firmware-level aware of NTFS (or other filesystems) and can adapt directly to the way they're used, this still can present a performance bottleneck as your files become fragmented.

When I allocate fixed-size, yeah, some fragmentation certainly can occur on the SSD, but it's far less because we're asking for the total size to be reserved away right off the bat. You'll definitely have a lot of contiguous block space on the SSD being used for the VDI. This means that your performance would be the same as if you had installed your OS directly on the SSD itself.

I hope that made sense.

Now, back to trying to clone this image....

Thanks

F

[mpack]

Now, if I use a dynamically allocating disk, the disk expands as I use sectors on it. The problem is these sectors will never be anywhere near aligned with the host filesystem's blocks. So say that I write 4KB to my virtual disk, and it expands on the SSD. Now, to write that 4KB, the SSD has to erase then write 128KB.

Are you aware that the dynamic VDI image is internally allocated in 1MB blocks? I.e. the unit of allocation is 1MB? The bit in the manual about it expanding when new sectors is written is kind of true, but also a simplification: in fact what happens is that the VDI allocates whichever 1MB block includes the written sector, if it hadn't done so already. This means that the unit of allocation of a VDI block would be >= the size of your suggested typical SSD erase block. Yes, it its still true that a Windows/Linux cluster is smaller, but that fact is inescapable regardless of fixed vs dynamic. The only practical difference is the potential for fragmentation at the 1MB block level (a problem which CloneVDI can fix btw), which brings back the lack of seek latency mentioned earlier.

[fmillion]

@mpack: Actually, no, I never did read that part of the manual. That's very good to know, actually. I might actually start using dynamic images on here after all. Thanks.

Either way, alas, no go on the CloneVDI method. I think I did figure out what's going on, I just can't figure out how to solve it.

The original large disk was 8.0GB, and the new disk is 3.6GB. My research on the NT boot process says that for all disks below 8GB, the NT loader code uses CHS addressing; for disks over 8.4GB it will use LBA. Also, it stated that "most" BIOSes will use variations of CHS translation, often based on the size of the disk in question.

This interests me because, when I first created the new blank disk image, I noted that fdisk was attempting to give it a very strange geometry - something like 204 heads, 39 sectors or something. I forced fdisk to use 255/63 because I figured this is what the old NT partition was based on. But I realized later that this won't change how the BIOS decides it wants to translate the disk into CHS. (Remember that it still didn't work even when I did leave the partition alone, at the odd CHS values.)

This in mind, I modified the NT partition boot sector to reflect the values Windows XP setup gave to the partition - it used 128 heads, 63 sects. This coincides with the documents I read that said that for disks between 2 and 4 GB, BIOSes will typically use 128 heads. (64 for 1-2GB, 32 for 512M-1G, and 255 for 4GB and up).

This still didn't fix the problem though. I still get the "disk read error occurred" message. I actually did find a few posts around the Internet that suggested this very fix for *expanding* a disk - i.e. the original NT partition was marked as being based on 128 heads, and since the disk was expanded now the BIOS was using 255 heads, so the boot sector needed to be modified to reflect that.

This makes me wonder if VBox's BIOS is doing something funky with CHS translation. I still haven't tried letting fdisk use that strange geometry then setting the boot sector to match it, and I might try that, but it seems like the BIOS would be being strange to pick values like that.

I'm almost at the point of reinstalling the new VM - but that's a pain in and of itself. Especially since I'm trying to fit the *finished* installation into a 3.6GB. IT's easy to fit the completed installation there, but nigh on impossible if I have to run Windows Update, program installers, etc. (In the original image, the large size allowed that, after which I cleaned out all the unnecessary crud such tasks leave behind, resulting in the slim,faster installation.)

So... Any more ideas?

Thanks

F

[vbox4me2]

What about recreating the disk you need with the size you want and then use a linux ISO or another XP vdi to copy disk1 to disk2 ? you might need to fdisk the new disk to get the boot loader working or just install xp on the new disk and overwrite it with described process, or ntbackup/restore between them.

[fmillion]

Found some more info...

Look at this from the log of the VM execution.

Code: Select all

00:00:06.422 Guest Log: BIOS: int13_harddisk: function 02, disk 80, parameters out of range 0187/00a0/0019!

I'm not sure but I assume it's reporting C/H/S, so it's trying to access head 0x00A0, which would be out of range. (Linux and Windows think the disk only has 128 heads, which is 0x0080!) Do note that I've already altered the partition boot sector to indicate only 128 heads. This is either being ignored or is not checked upon boot.

Now, I just have to figure out why it insists on doing that. I don't think installing Windows then cloning the disk back to the same image will make any difference, as ntfsclone will overwrite everything anyway...

F

[fmillion]

Ok, so I finally figured out the booting part. I'm not exactly sure WHY what I did worked, but it did.

I first started by erasing the disk and recloning. Then, I ran the "Testdisk" application under Linux to repair the NTFS boot sector, after informing testdisk that I want it to assume 128 heads. This made Windows choke on the NTFS, so I booted the XP CD and ran Chkdsk from recovery console. After that... It booted!

I have a more complicated problem now though which might mean I have to start over (but at least I learned.) I foolishly used the SAME virtual machine for the cloning that I used to BOOT to test the disk. (It's hard to explain...) Basically, I have a new VM, with a C drive (the large disk.) I created the second (smaller) disk, attempted all the cloning, etc. but nothing worked. Then, I booted the still working (larger) image and ATTACHED the smaller image. This meant that NTFS assigned it "D:" Later, when things failed further, I cloned the larger disk again to the smaller image. Remember that by now the OS on the larger disk has seen the smaller disk as D:. So now when XP tries to boot FROM the smaller disk, I'm guessing it's connecting the itself to drive D:, which obviously won't work... and I'm getting a boot loop.

Oh well... Next time I'll know better. I'll start over with the original machine and make a whole new clone and start from there. If anyone could help me understand exactly what went wrong (And why TestDisk fixed it) though I'd be grateful... The more I understand the better I am at doing this stuff.

F

[mpack]

It's probably a bad idea to cross the 8GB boundary unnecessarily. CloneVDI knows to patch the drive geometry info when it enlarges a drive from below 8GB to above it, but of course CloneVDI doesn't allow you to shrink a disk, so the opposite case does not arise.

Frankly, I think you are making life difficult for yourself for no good reason. Just clone the original drive (assuming you don't want to modify the original). Then use gparted to shrink the partition, then clone it again using CloneVDI, with the compact option enabled. You will now have a dynamic drive whose partition size is limited to whatever you set, and where areas of the drive not used by the partition are discarded from the image. The drive size is irrelevant.

[fmillion]

I did finally make everything work.

Here's the final steps I followed:

1. Created a brand new clone. (The clone I had been working with was too far gone.) Prepared the OS as I'd done in the first place.
2. Created the new smaller disk image.
3. Booted to Linux from CD and partitioned the new image, forcing fdisk to use 128 for the heads value in CHS.
4. Ntfsresize'd the partition on the large disk, then Ntfsclone'd the NTFS partition from the larger disk to the smaller disk.
5. Ntfsresize'd the smaller disk's NTFS to fill the partition
6. Ran TestDisk and informed it of the 128 heads geometry, and had it repair the NTFS boot sector.
7. Disconnected the larger disk, reassigned the smaller one to primary master.
8. Booted the XP recovery console to install a fresh MBR. (I could probably also have done this by dd'ing 446 bytes, but I did it this way just to make sure.)
9. Everything worked. Windows chkdsk'ed the volume upon boot, fixed some errors (I don't know exactly what), and rebooted and Windows came up running off the smaller image.

@mpack: You're right. I'm probably way over doing this. As a tech hobbyist though I tend to like to try things, experiment, tinker around and just find out what can and can't work. While this probably wasn't necessary for the final goal, I feel I learned something about VBox, disk images, NTFS, etc. in the process, so it was worth it.

Based on what you said about CloneVDI changing the disk geometry when you expand an image...Does this mean the VDI image itself contains the disk geometry that the BIOS will present to the virtual machine? (That makes me wonder if there's some cool command line options on, say, vboxmanage, to create images with custom CHS values...!

Thanks again everyone for your input!

F

[mpack]

Yes, the VDI header includes a disk geometry: cylinders, heads and sectors per track (aka CHS). I've not checked the source code but I assume this is the geometry which the virtual IDE controller will report to the BIOS. The boot sector of a Windows partition (not to be confused with the MBR) also contains a "#heads" field which matters if the CHS addressing scheme hasn't maxed out.

Btw, the reason for the 8GB threshold is that the MBR has a fixed number of bits for each field: 10 bits for C, 8 bits for H and 6 bits for S, so max values are 1023,255,63. Some of these are offset by 1, though offhand I cant remember which. If you multiply out these limits assuming 512 bytes per sector then you get something around 8GB. If you were to fiddle with funny CHS values then you would need to remember these limits. However I'm not aware of an easy way for you to fiddle.