Problem with snapshots
-
- Posts: 5
- Joined: 23. Aug 2021, 07:08
Problem with snapshots
Sadly this is not a one off.
Many years ago I automated snapshot backups.
Today I found sereval instances of the same VMs running, and the disk images (VDIs) are destroyed.
Litterally stomped to pieces.
Can't even convert to raw and get any file system recognised.
My guess is the differencing vdi had failed and then got merged back into the main vdi, during the automatic snapshot deleting process.
I saw this one other time - and managed to recover to prior snaps.
Oh well - time to switch to KVM.
Many years ago I automated snapshot backups.
Today I found sereval instances of the same VMs running, and the disk images (VDIs) are destroyed.
Litterally stomped to pieces.
Can't even convert to raw and get any file system recognised.
My guess is the differencing vdi had failed and then got merged back into the main vdi, during the automatic snapshot deleting process.
I saw this one other time - and managed to recover to prior snaps.
Oh well - time to switch to KVM.
Last edited by mpack on 23. Aug 2021, 09:56, edited 3 times in total.
Reason: Change title to something appropriate.
Reason: Change title to something appropriate.
-
- Site Moderator
- Posts: 39134
- Joined: 4. Sep 2008, 17:09
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Mostly XP
Re: VirtualBox starts SAME virtual machine multiple times
I don't see what your snapshots problem has to with the OP's "VM starts multiple times" problem. I'm going to split your topic off.
What you seem to be saying is that you wrote some scripts, and the lack of error checking in those scripts has resulted in a damaged VM. I doubt that KVM will solve this problem for you.
Talking about solutions, I don't suppose you had some actual backups, to secondary storage? If you have that then this is nothing but a minor inconvenience - i.e. it's the whole reason we make backups, and if your backup strategy can't fix your problem in this way then... well now you know why we keep saying it: snapshots are not backups.
Assuming that the answer is no, then you may be able to recover something using CloneVDI: look at the readme instructions on cloning snapshots. If you can clone a newish snapshot then you'll have recovered all of your data to that point in time. On a Linux host you'll have to run CloneVDI under Wine. Or you can copy all the files to a Windows host.
In fact you didn't, because there's no such thing as a "snapshot backup". Snapshots are the opposite of backups.Dacronious wrote: Many years ago I automated snapshot backups.
Nor is there an automatic snapshot deleting process.Dacronious wrote: My guess is the differencing vdi had failed ... during the automatic snapshot deleting process.
What you seem to be saying is that you wrote some scripts, and the lack of error checking in those scripts has resulted in a damaged VM. I doubt that KVM will solve this problem for you.
Talking about solutions, I don't suppose you had some actual backups, to secondary storage? If you have that then this is nothing but a minor inconvenience - i.e. it's the whole reason we make backups, and if your backup strategy can't fix your problem in this way then... well now you know why we keep saying it: snapshots are not backups.
Assuming that the answer is no, then you may be able to recover something using CloneVDI: look at the readme instructions on cloning snapshots. If you can clone a newish snapshot then you'll have recovered all of your data to that point in time. On a Linux host you'll have to run CloneVDI under Wine. Or you can copy all the files to a Windows host.
-
- Posts: 5
- Joined: 23. Aug 2021, 07:08
Re: Problem with snapshots
"I don't see what your question has to do with" "VM starts multiple times"
Let me explain; Each VM was running 2 or 3 times. I automated snapshots, and used them as backups - it has worked very well to get around malware and windows update issues.
When the snapshotting happened (nightly) with multiple instances of the same VMs running - it corrupted the VM's vdi - it was so irreparable that attempting to boot, ended with "missing operating system"
I automated the snapshot deletion process (Nightly) as well - I'd say the differencing VDI was rubbish, but it got merged back into the VM's base VDI.
I wrote some scripts - Yes - they worked for about 12 years. I doubt there are errors. However the VirtualBox service managed to screw up and start the VMs multiple times. THIS is what caused the failure, not the scripts.
Backups in secondary storage - is what saved anything - (Also half hourly offsite critical data backups) - but I did not expect the VirtualBox Svc to properly trash the VDI files to the point that there was nothing recognisable - not even the disk partitions.
I know how to run this software - I know how to run backups - I am highlighting that there is a critical fault - adding onto a post where someone quipped "this is a one off".
If this is the attitude that is presented towards critical faults; I wont be looking back after moving to KVM.
Let me explain; Each VM was running 2 or 3 times. I automated snapshots, and used them as backups - it has worked very well to get around malware and windows update issues.
When the snapshotting happened (nightly) with multiple instances of the same VMs running - it corrupted the VM's vdi - it was so irreparable that attempting to boot, ended with "missing operating system"
I automated the snapshot deletion process (Nightly) as well - I'd say the differencing VDI was rubbish, but it got merged back into the VM's base VDI.
I wrote some scripts - Yes - they worked for about 12 years. I doubt there are errors. However the VirtualBox service managed to screw up and start the VMs multiple times. THIS is what caused the failure, not the scripts.
Backups in secondary storage - is what saved anything - (Also half hourly offsite critical data backups) - but I did not expect the VirtualBox Svc to properly trash the VDI files to the point that there was nothing recognisable - not even the disk partitions.
I know how to run this software - I know how to run backups - I am highlighting that there is a critical fault - adding onto a post where someone quipped "this is a one off".
If this is the attitude that is presented towards critical faults; I wont be looking back after moving to KVM.
-
- Site Moderator
- Posts: 39134
- Joined: 4. Sep 2008, 17:09
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Mostly XP
Re: Problem with snapshots
These are the VirtualBox user discussion forums. None of us has any vested interest in the software you choose. If you prefer KVM, that's great, there's no need to keep mentioning it.Dacronious wrote:I wont be looking back after moving to KVM.
How can "critical faults" exist in features that do not exist? (see the first two sentences of my previous reply). The claim is nonsensical. It was your inadequate scripts which failed - the fact that they did not fail in the past (when there were no system faults to deal with) really says very little about their quality.
You say you know how to make backups, but the facts appear to indicate otherwise: if you know how to make backups then you can't have lost data. Unless you are saying that you have in fact recovered your data using a real backup and all of the above is just for our FYI?
-
- Site Moderator
- Posts: 20945
- Joined: 30. Dec 2009, 20:14
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Windows, Linux
Re: Problem with snapshots
I would like to add one thing:
In using Virtualbox myself, both in home and automated office environments, and for some years IT'ing the office server running business-critical Virtualbox VMs, and in ten years of reading the forums I have not heard of Virtualbox spontaneously starting a VM.
However, when self-rolled scripts are being used, all bets are off. I would far more easily diagnose the problem you had, based on the little evidence presented, as an unexpected failure in the scripts, perhaps in not enough condition testing or error-checking. An unexpected set of circumstances occurred that the scripts were not designed to test for, and the scripts restarted things, apparently multiple times. Since we don't see the scripts we cannot diagnose this problem, as it simply does not happen that Virtualbox spontaneously start VMs.
Now for what it is worth, Virtualbox has failed to properly handle a snapshot before, if that is what happened here. Note that this cannot be stated for certainty, because (the scripts caused?) the VM starting multiple times. Errors happen, it is a computer, after all. Thus the forum's constant mantra: Snapshots make a VM more delicate, and the VM needs to be properly backed up in a reliable method. The forum has also pointed out consistently that snapshots are not backups, and that a backup routine using snapshots cannot be thoroughly relied upon. (Even though it seemed to work in your case, the data is not reliable. If you want to know why, please post back.)
In using Virtualbox myself, both in home and automated office environments, and for some years IT'ing the office server running business-critical Virtualbox VMs, and in ten years of reading the forums I have not heard of Virtualbox spontaneously starting a VM.
However, when self-rolled scripts are being used, all bets are off. I would far more easily diagnose the problem you had, based on the little evidence presented, as an unexpected failure in the scripts, perhaps in not enough condition testing or error-checking. An unexpected set of circumstances occurred that the scripts were not designed to test for, and the scripts restarted things, apparently multiple times. Since we don't see the scripts we cannot diagnose this problem, as it simply does not happen that Virtualbox spontaneously start VMs.
Now for what it is worth, Virtualbox has failed to properly handle a snapshot before, if that is what happened here. Note that this cannot be stated for certainty, because (the scripts caused?) the VM starting multiple times. Errors happen, it is a computer, after all. Thus the forum's constant mantra: Snapshots make a VM more delicate, and the VM needs to be properly backed up in a reliable method. The forum has also pointed out consistently that snapshots are not backups, and that a backup routine using snapshots cannot be thoroughly relied upon. (Even though it seemed to work in your case, the data is not reliable. If you want to know why, please post back.)
-
- Posts: 5
- Joined: 23. Aug 2021, 07:08
Re: Problem with snapshots
These responses highlight to me that no one is interested in improving the software.
VirtualBox allows multiple instances of the same VM to run. This is OBSERVED. This is not 'A ONE OFF'.
Instead of addressing 'how my backups were insufficient' - how's about addressing or even acknowledging the problem?
Are you people able to see that running the multiple instances of the same virtual machine is A BAD THING?
And that the software ALLOWS THIS TO HAPPEN?
The only reason I am posting here is to warn people that this can happen - and VirtualBox has regressed and is not production ready. I have tested this previously (years and versions ago) and this could not happen.
VirtualBox allows multiple instances of the same VM to run. This is OBSERVED. This is not 'A ONE OFF'.
Instead of addressing 'how my backups were insufficient' - how's about addressing or even acknowledging the problem?
Are you people able to see that running the multiple instances of the same virtual machine is A BAD THING?
And that the software ALLOWS THIS TO HAPPEN?
The only reason I am posting here is to warn people that this can happen - and VirtualBox has regressed and is not production ready. I have tested this previously (years and versions ago) and this could not happen.
-
- Site Moderator
- Posts: 20945
- Joined: 30. Dec 2009, 20:14
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Windows, Linux
Re: Problem with snapshots
Please provide evidence that virgin(*) Virtualbox can start a VM more than once.
(* meaning not influenced by user-made scripts. Virtualbox Manager or manually-typed vboxmanage commands only.)
(* meaning not influenced by user-made scripts. Virtualbox Manager or manually-typed vboxmanage commands only.)
-
- Posts: 5
- Joined: 23. Aug 2021, 07:08
Re: Problem with snapshots
Am I to understand correctly,
that if 'vboxmanage' manage is invoked by a bash script it behaves differently than if it invoked from a tty?
If so, can you please show me where that is advertised in the documentation?
that if 'vboxmanage' manage is invoked by a bash script it behaves differently than if it invoked from a tty?
If so, can you please show me where that is advertised in the documentation?
-
- Site Moderator
- Posts: 39134
- Joined: 4. Sep 2008, 17:09
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Mostly XP
Re: Problem with snapshots
Don't be disingenuous. You know very well what is being said. A human user will (hopefully) not ignore errors.
I suggest that you stop trying to point fingers and pass the buck. You are entitled to nothing. If you have a concrete proposal, or source patches, then by all means please expand on your solution.
I suggest that you stop trying to point fingers and pass the buck. You are entitled to nothing. If you have a concrete proposal, or source patches, then by all means please expand on your solution.
-
- Posts: 5
- Joined: 23. Aug 2021, 07:08
Re: Problem with snapshots
This is conversation is getting weirder and weirder.
How am I 'passing the buck'?
I've used Virtualbox for over 12 years, deployed it on many occasions, on many sites. It was my go to hypervisor of choice. Now I've had to re-evaluate that. I don't want to particularly learn a new VM ecosystem, however I can't trust this system as it completely destroyed the VDIs that were attached to it.
You blame scripts - however - this is the best example of burying one's head in the sand. I piggy backed another post, where someone else reported what I saw, and was dismissed as having 'a one off' situation. It was not a one off.
Answer this question; under what circumstances should a VM start twice? Why isn't this situation adequately protected from happening? Who cares if it was automated scripts that made it trip over and royally fail?
How am I being 'entitled'?
I am explaining that there is a critical issue - that I have lost years of work over. I am reporting an issue, which the software allows itself to get into an absurd state which eventually becomes destructive (if left to fester)
"Ignore errors"?
Please explain what it the expected output of the VBoxManage command when it happily accepts a directive to spin up a second version of the same VM.
How am I 'passing the buck'?
I've used Virtualbox for over 12 years, deployed it on many occasions, on many sites. It was my go to hypervisor of choice. Now I've had to re-evaluate that. I don't want to particularly learn a new VM ecosystem, however I can't trust this system as it completely destroyed the VDIs that were attached to it.
You blame scripts - however - this is the best example of burying one's head in the sand. I piggy backed another post, where someone else reported what I saw, and was dismissed as having 'a one off' situation. It was not a one off.
Answer this question; under what circumstances should a VM start twice? Why isn't this situation adequately protected from happening? Who cares if it was automated scripts that made it trip over and royally fail?
How am I being 'entitled'?
I am explaining that there is a critical issue - that I have lost years of work over. I am reporting an issue, which the software allows itself to get into an absurd state which eventually becomes destructive (if left to fester)
"Ignore errors"?
Please explain what it the expected output of the VBoxManage command when it happily accepts a directive to spin up a second version of the same VM.
-
- Site Moderator
- Posts: 20945
- Joined: 30. Dec 2009, 20:14
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Windows, Linux
Re: Problem with snapshots
I ask for evidence, you still respond with unproven assertions.Dacronious wrote:How am I 'passing the buck'?
No evidence presented yet that Virtualbox started a VM multiple times. One more chance then this topic gets locked.