Accidentally detached snapshot

Discussions related to using VirtualBox on Linux hosts.
Post Reply
VisVask
Posts: 5
Joined: 21. Jul 2019, 13:30

Accidentally detached snapshot

Post by VisVask »

Hi, guys. I really need some help!


I don't know if my VM is salvageable anymore, but if you have any suggestions, please let me know.

Ok. So here is what happened:

1. My host machine ran out of space. It has a 2TB hard drive.

I was running two VMs on the host.

VM1: 400GB fixed storage and one snapshot
VM2: 800GB fixed storage and one snapshot

VM2 was running Nextcloud (a self hosted cloud solution otherwise much like Google Drive). All the cloud files was being stored on a separate 4TB harddrive.

What I didn't know was that the main hard drive would fill up while Nextcloud was copying files to the external drive. I guess this is because of the snapshot, which grew really big and ended up consuming the main drive. The VM2's Snapshot grew to 553GB before the host system ran out of space and paused the guest VMs.

I tried to delete the snapshots to free up space, but that apparently required a lot of free disk space, which i didn't have.

My solution:
What I ended up trying, was moving my Snapshots folders over to the external drive and then putting a symlink in their original location.

This seemed to work fine. When I started up VirtualBox everything looked fine again (with some minor hiccups that involved renaming `...-prev` files), so I deleted the VM1's snapshot. After the snapshot was merged, I could finally boot it up. It was fine!

So I proceeded to delete VM2's snapshot. It wasn't there?! So, dumb as I am, I tried to boot it up. `FATAL: Could not read from the boot medium! System halted`. After a bit of research, I realised that the VM's drives was missing..? But I just readded the `.vdi` file and tried to boot up again. `Out of memory` (or something). Only then I realised that all of the VM's settings was reset to default.

Not knowing what else to do, I reassigned it 3GB of ram and 12MB of VRAM and booted it up again. It worked! Or so I thought. I had booted up into the year old snapshot! Without doing anything, I quickly shut the VM down again.

The problem:
VM2 is working, but I have apparently lost a year worth of important work and data. I have all the VM files, but I am worried that booting up the VM changed it, so that it's no longer possible to merge it with the snapshot. Is this the case or are there any way to extract my data from the snapshot?

Also, what happens if I swap the current harddrive `.vdi` with the snapshot's `.vdi`?

Structure:
VM2 (VM2's name is `Octopus`) folder:
drwx------ 2 root root 4,0K juli  20 18:49  Logs
-rw------- 1 root root 2,9K juli  20 18:53  Octopus.vbox
-rw------- 1 root root 2,9K juli  20 18:53  Octopus.vbox-prev
-rw------- 1 root root 801G juli  20 18:53  Octopus.vdi
-rw-r--r-- 1 root root 1,0K juli  20 19:05  .Octopus.vdi.swp
lrwxrwxrwx 1 root root   48 juli  20 18:05  Snapshots -> /mnt/[external drive]/[path to snapshot]
Snapshot folder:
-rw------- 1 root root 1,4G april 29  2018 2018-04-29T19-21-23-821307000Z.sav
-rw------- 1 root root 553G juli  20 12:20 {39######-####-####-####-######667e96}.vmdk
-rw------- 1 root root  26G juli  20 12:20 {95######-####-####-####-######220193}.vdi
Related:
After my hours of googleing I found this related issue (which didn't help me): ...cannot post urls yet... But it was the second result after googleing for `virtualbox detached snapshot` (View topic - accidentally "released" my snapshot ...)

I also tried to merge my snapshot with this app: ...Can't post urls yet. It's a VDI cloning tool posted on these forums (CloneVDI tool - Discussion & Support)

But I got an error saying `Source is a different image. Sorry, this tool cannot clone these (for now)`. But I may have misunderstood something.
Last edited by VisVask on 3. Sep 2019, 20:08, edited 1 time in total.
VisVask
Posts: 5
Joined: 21. Jul 2019, 13:30

Re: Accidentally detached snapshot

Post by VisVask »

Can anyone please point me in the right direction? Any help would be greatly appreciated!
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Accidentally detached snapshot

Post by socratis »

And that's why you got to start by experimenting first with a feature, before implementing it, to understand how it works and what you're supposed to do/don't. I know, I sound like your parents when they would say "I told you to not play with fire"... :)

I need to see the .vbox of the VM, ZIP it and attach it (see the "Upload attachment" tab below the reply form). I also need to see the complete paths. The "ls -alR /mnt/[external drive]/[path to snapshot]", obviously not obfuscated or altered at all.

Also, why is there a .vmdk snapshot along with a .vdi snapshot, both at the same date and time? The size discrepancy is huge to make sense.

In the meantime, please read the Snapshot basics post, it will help you answer the question:
VisVask wrote:I have all the VM files, but I am worried that booting up the VM changed it, so that it's no longer possible to merge it with the snapshot. Is this the case or are there any way to extract my data from the snapshot?
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
VisVask
Posts: 5
Joined: 21. Jul 2019, 13:30

Re: Accidentally detached snapshot

Post by VisVask »

Hi, Socratis!

I'm very sorry for my late reply. I didn't realize someone had replied till last week, a day before I was going on a short vacation, and was regrettably unable to reply right away.

However, I'm back now, and very much appreciate any time you can spare to help me. Although I guess I might be a lost cause.

You are totally right, by the way. I shouldn't have used the snapshot feature in "production" before I'd studied it more. However, when I created the snapshot I was really deep in the weeds and made it as a backup plan while trying to recover the VM. You see, the VM seemed to be corrupt after the host machine had shut down unexpectedly during a thunderstorm. I was unsure if I could fix it, and made the snapshot so I could go back in case I screwed something up while fixing it.

When I had recovered the VM, I was going to delete the snapshot, but the snapshot deletion warning was kind of scary and I didn't make time to read the docs, so I just left it (I guess we've all been there? :P ). Also, I figured a snapshot might come in handy if I messed up the VM later, so instead I made snapshots of the other machines too. I guess it's kind of ironic that it was the snapshots that messed it all up... :shock:

I read your article about snapshots basics (good article, by the way), and from what I gather, my files are lost if booting it up actually changed it. Which it surely did?

Here is the complete paths, as requested:

Code: Select all

root@Aquarium:/mnt/Noah-akv/crysisManagement/Octopus/Snapshots# ls -alR /mnt/Noah-akv/crysisManagement/Octopus/Snapshots/
/mnt/Noah-akv/crysisManagement/Octopus/Snapshots/:
total 633521612
drwx------ 2 aquarium aquarium         4096 juli  21 14:52 .
drwxr-xr-x 4 root     root             4096 juli  21 12:18 ..
-rw------- 1 aquarium aquarium   1406675594 april 29  2018 2018-04-29T19-21-23-821307000Z.sav
-rw------- 1 aquarium aquarium 592838459392 juli  20 12:20 {396e971b-af56-4e84-b7e7-72cbb2667e96}.vmdk
-rw------- 1 aquarium aquarium  27435991040 juli  21 16:14 {95162384-429a-420a-9538-fffba5220193}.vdi
-rw------- 1 root     root      27433893888 juli  21 14:59 {95162384-429a-420a-9538-fffba5220193}.vdi-2
(In my last post I had replaced the VM's real name with `VM2` to make it easier, but it's real name is `Octopus`. I realize now that that might cause more confusion then it would omit, so I'll edit my first post to change it back. I guess you would appreciate that.)

The .vbox is zipped and attached. Although I renamed it, as I got an upload error saying: `The extension vbox is not allowed.`. (The original filename was `Octopus.vbox`).
Attachments
Octopus.vbox.txt
(3.31 KiB) Downloaded 19 times
VisVask
Posts: 5
Joined: 21. Jul 2019, 13:30

Re: Accidentally detached snapshot

Post by VisVask »

I guess I won't be able to recover my data, so I'll resume operations from the old version of the VM in a couple of days, unless anyone of you tell me you think it's recoverable..?

Also, thank you Socratis! I guess you couldn't help me this time, but I see you are a big contributor to the forum – keep it up! :D


Have a nice day, folks!
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Accidentally detached snapshot

Post by mpack »

Sorry, I seem to have overlooked this topic until now.

Fixed size storage plus a snapshot...? I wonder what you thought you were doing there? People use fixed because they read some dolt saying it gives better performance (in fact the opposite is true). A more legitimate reason is for a high availability server VM to ensure that the VDI can never run out of host disk space, because it's all pre-allocated. Choosing fixed is already a questionable decision, then creating a snapshot on top of that completely invalidates even the bad justifications for it.

Anyway, there's no way for a non-expert to re-attach a snapshot. The normal way to recover is to merge them instead. Put all the VDI files (base plus snapshots) in one folder and clone the last snapshot using CloneVDI. Since you have a Linux host you'll have to run CloneVDI under Wine. The resulting clone is a dynamic VDI containing a merge of base+snapshots. You must build a new VM around the clone - it isn't usable in the original snapshot-expecting VM.
VisVask
Posts: 5
Joined: 21. Jul 2019, 13:30

Re: Accidentally detached snapshot

Post by VisVask »

Hi, mpack, thanks for reaching out!

I'm sorry for taking so long to come back to you, but the merging takes around 10-15 hours and I tried three times. They errored out all three times.

The error showed up at 68% all three times, which leads me to believe that there is something in my files that makes them impossible to merge.

The error message says:
Got OS error 0 when reading from source file
Is there a way to fix this or is this?

In CloneVDI I used the `/Snapshots`-folder as source and `/snap-merge` as the destination.

Code: Select all

aquarium@Aquarium:/mnt/Noah-akv/crysisManagement/Octopus$ ls -alhR
.:
total 16K
drwxr-xr-x 4 root     root     4,0K sep.   8 18:11 .
drwxr-xr-x 5 root     root     4,0K juli  20 19:01 ..
drwxr-xr-x 2 aquarium aquarium 4,0K sep.  10 19:04 snap-merge
drwx------ 2 aquarium aquarium 4,0K sep.   8 18:11 Snapshots

./snap-merge:
total 6,3G
drwxr-xr-x 2 aquarium aquarium 4,0K sep.  10 19:04  .
drwxr-xr-x 4 root     root     4,0K sep.   8 18:11  ..
-rw-r--r-- 1 aquarium aquarium 6,3G sep.  11 01:28 'Clone of Octopus.vdi'

./Snapshots:
total 605G
drwx------ 2 aquarium aquarium 4,0K sep.   8 18:11 .
drwxr-xr-x 4 root     root     4,0K sep.   8 18:11 ..
-rw------- 1 aquarium aquarium 1,4G april 29  2018 2018-04-29T19-21-23-821307000Z.sav
-rw------- 1 aquarium aquarium 553G juli  20 12:20 {396e971b-af56-4e84-b7e7-72cbb2667e96}.vmdk
-rw------- 1 aquarium aquarium  26G juli  21 16:14 {95162384-429a-420a-9538-fffba5220193}.vdi
-rw------- 1 root     root      26G juli  21 14:59 {95162384-429a-420a-9538-fffba5220193}.vdi-2
lrwxrwxrwx 1 root     root       40 sep.   8 18:01 Octopus.vdi -> '/root/VirtualBox VMs/Octopus/Octopus.vdi' 
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Accidentally detached snapshot

Post by mpack »

"OS Error 0" means unexpected end of file. That means it wanted to continue reading, but there was no more data in the file, because the file was truncated somehow.

There's no fix for that. All you can do is clone the next oldest snapshot instead.

I see you have a weird mix of vdi and vmdk snapshots. I don't use snapshots so I have no idea how that could ever happen. I also see one vdi and one "vdi-2" with the same UUID. What is that? VDI doesn't support split files, and I can't think what other purpose they could serve.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Accidentally detached snapshot

Post by socratis »

Lots of issues here...
VisVask wrote:lrwxrwxrwx 1 root root 40 sep. 8 18:01 Octopus.vdi -> '/root/VirtualBox VMs/Octopus/Octopus.vdi'
That's a link to the actual VDI, which is under root's account? So, you have a VM in an external drive (/mnt/Noah-akv/crysisManagement/Octopus) with the VDI being a ... link?
[color=#00AA00]mpack[/color] wrote:I see you have a weird mix of vdi and vmdk snapshots
I don't see any evidence of that VMDK being used anywhere in the VBOX file. And obviously I have no clue how/why it exists, it serves absolutely no purpose. Maybe the OP tried to clone something as a VMDK?
[color=#00AA00]mpack[/color] wrote:I also see one vdi and one "vdi-2" with the same UUID. What is that?
I believe it might actually be a straight out file copy from within the OS. The VDI is "aquarium:aquarium" while the VDI-2 is "root:root".

NOTE TO "VisVask"
There's absolutely no reason to be 'root', on the contrary, it's strongly advise not to! VirtualBox offers you absolutely no advantage while being 'root', it won't even show you the VMs that you've registered as "aquarium".
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Post Reply