Storing VDI on cloud drive

[Joe Enos]

Does anyone have any experience storing the VDI file on a cloud drive? Specifically, I'm looking into using Bitcasa infinite storage, which stores data on the cloud only, and maps the cloud drive so you can see it like a regular disk (there's local caching and cool stuff like that, but the file basically lives on the cloud). If not Bitcasa, then DropBox or SkyDrive or other similar services.

I'd like to try this so I can have lots of virtual machines without using local space, but I'm concerned about the performance of accessing a remote filesystem to read and write the VDI. I'm thinking that it likely won't work well, because of all the bandwidth required to download and sync the VDI file. But at the same time, I don't know anything about the VDI file structure - if it stays relatively stable and doesn't change a whole lot while it's running, then Bitcasa should be smart enough to only transfer the changed pieces, rather than the entire file every time. But if the file is constantly moving bits around on disk as I'm working, then I'd expect it would be practically impossible, or use a ridiculous amount of bandwidth.

Bitcasa says they've never done any official testing, but expects that it should work as long as my connection is good enough.

Any stories or theories on this? I'm planning on trying it with a small VM, but I don't want to accidentally waste a TB of bandwidth attempting a larger one if everyone has been unsuccessful at it.

[mpack]

I expect that the drive would be unusably slow. What kind of throughout might you expect from an internet connection? Unless you're on a dedicated gigabit network I'd guess you'd be lucky to get 4MB/s. Even 1MB/s (8mbps) is considered good in many places. By comparison, your typical local hard disk shifts around 25MB/s. Caching won't help much. Can you live with a >25x loss of performance? I doubt I could.

Note that this is all assuming that the connection is working perfectly, then you'd get an unusably slow drive. If the connection was less than perfect, glitchy etc... I shudder to think what the experience might be like.

[Joe Enos]

Thanks, I expected this would be the case. It makes sense, assuming I'd be using the physical disk a lot. I guess it comes down to how the physical disk is accessed, and how the cloud disk provider manages the local cache.

For example: suppose I have a 20GB VDI file with a Windows XP guest. The VDI file lives on Bitcasa's infinite drive. Obviously in order to start the machine I'd have to download enough of the VDI to boot to Windows and display the desktop, as well as to run the startup services, write to the event log, etc. - but I'd expect that I wouldn't need to download the entire 20GB - assuming Bitcasa's partial file download and caching works well, and the guest knows exactly which pieces of the VDI it needs, it may only take (I'm pulling numbers out of the air here) 100MB of actual data transfer to start things up, which wouldn't be terrible.

If I create a new 200KB file on my guest disk, that should add/modify 200KB of physical bits on the VDI file. Assuming the remainder of the VDI file remains unchanged, and again assuming Bitcasa's partial file upload works well, that would only mean about 200KB would need to be uploaded to the cloud, and assuming Bitcasa's local caching and queueing works properly, that upload would be deferred to the point where I wouldn't even see the delay. Obviously the more files that get modified, and the larger the files are, the bigger this process becomes, which is where a small test would differ dramatically from a larger-scale test.

I'm making a lot of assumptions that I don't know if they're even close to accurate, which is really why I'm asking if anyone has ever tried this on a larger scale.

From the VirtualBox side, I guess I can run some tests on the VDI file itself while I have a guest OS running, and see how much the file changes as I perform minor tasks on the guest. Then it would come down to how fast and how accurate Bitcasa would be detecting and uploading these changes.

Thanks for your response.

[mpack]

From the VirtualBox side, I guess I can run some tests on the VDI file itself while I have a guest OS running, and see how much the file changes as I perform minor tasks on the guest.

No need for that, VDI file's are an open book, the source code being available for examination.

Although technically it can vary, all VDIs so far use a block size of 1MB. That means that if you write to a new 1MB "page" of the disk then a whole 1MB is allocated to the host file (I'm assuming dynamic VDI here - a fixed say VDI would take way too long to initialize in a remote storage). However once the block is allocated any mods to it will thereafter work as you guessed, with change granularity being dictated by the block/cluster size of the guest OS (typically 4K).

I'm surprised that you're so sanguine about having to download 100MB from the web every time you want to boot up a VM. I live in a rural location and that would be a totally impractical proposition - and undesirable too if I had 3G/4G data caps to think about.

I assume the goal is portability. Is there some reason you can't just keep the VDI on a USB flash drive? That still wouldn't be as fast as a local hard drive, but it would be way faster than typical Internet speeds.

[Joe Enos]

Thanks for explaining the VDI format - I don't know much about low-level filesystem stuff in general, so I appreciate the explanation.

With a 30Mbps (~7.5MBps)

Edit: (3.75MBps - Wow, I really need to pay attention to my math)

DSL connection, 100MB is only about 15-30

Edit: 30-60

seconds assuming things are at their peak, so that's not a bad boot time. For 4G, I agree that it really would be ridiculous with both speed and data usage.

I think you're probably right on the better solution though - rather than trying to do something weird like this, I may just be better off with a portable hard disk (USB might be a little small, unless I shell out a bunch of money for large sticks). I do want to access this from multiple machines, in addition to the fact that I've been desperately running out of disk space on my laptop. Cloud storage was my first thought for convenience, but a portable disk might be my best alternative.

Thank you for your help.

[Perryg]

Cloud storage was my first thought for convenience, but a portable disk might be my best alternative.

If you have eSATA available on the laptop that is what I would suggest. My guests are stored on eSATA so I can run them at near native speed. (54MB/s) Oh and I would stay away from the so called green drives. Very poor performance IMHO.
1TB eSATA drives are fairly cheep these days.

[g8597963]

Thanks for explaining the VDI format - I don't know much about low-level filesystem stuff in general, so I appreciate the explanation.

With a 30Mbps (~7.5MBps) DSL connection, 100MB is only about 15-30 seconds assuming things are at their peak, so that's not a bad boot time. For 4G, I agree that it really would be ridiculous with both speed and data usage.

I think you're probably right on the better solution though - rather than trying to do something weird like this, I may just be better off with a portable hard disk (USB might be a little small, unless I shell out a bunch of money for large sticks). I do want to access this from multiple machines, in addition to the fact that I've been desperately running out of disk space on my laptop. Cloud storage was my first thought for convenience, but a portable disk might be my best alternative.

Thank you for your help.

Why not just leave your home PC on all the time and remote desktop to it in order to access the machine or pay for a hosting and do it on an actual server?

[Joe Enos]

If you have eSATA available on the laptop that is what I would suggest. My guests are stored on eSATA so I can run them at near native speed. (54MB/s) Oh and I would stay away from the so called green drives. Very poor performance IMHO.

Thank you, I'll keep that in mind.

[Joe Enos]

Why not just leave your home PC on all the time and remote desktop to it in order to access the machine or pay for a hosting and do it on an actual server?

RDC into a machine and using that machine to run a virtual machine might work - I've had issues in the past with double-hops like that, but it's definitely a possibility.

Hosting a cloud server is also possible - cost was one reason, but the more I think about it, it's not bad at all. I'd probably need a large EC2 or Azure instance, which may run 50 cents per hour or so, plus a few bucks a month for storage - but I don't plan on keeping it alive more than a few hours at a time anyway, so 10 or 20 dollars a month isn't unreasonable.

Thanks for the suggestions.