Beta 2: couldn't restore the saved state.

Temporary forum to discuss issues and problems during a VirtualBox Beta or Release Candidate phase.
Post Reply
Air Force One
Posts: 105
Joined: 6. Oct 2017, 16:54
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Windows
Location: Germany

Beta 2: couldn't restore the saved state.

Post by Air Force One »

Hi,

Today, I saved the state of my Windows 11 VM. But I couldn't restore it, because every time I try to do so, Vbox shows me an exception dialog and restoring process stopped.

I put my logs here. Maybe it would help to find the source of the troubles.
Attachments
Vbox_logs_crsh_restore_24092022.7z
Logs of VM
(45.36 KiB) Downloaded 58 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Beta 2: couldn't restore the saved state.

Post by fth0 »

VBox.log file wrote:
00:00:06.411490 VirtualBox VM 7.0.0_BETA2 r153583 win.amd64 (Sep 12 2022 13:28:42) release log
00:00:06.517811   Oracle VM VirtualBox Extension Pack (Version: 7.0.0_BETA2 r153583; VRDE Module: VBoxVRDP; Crypto Module: VBoxPuelCrypto)
00:00:12.400974 VMMDev: Guest Additions information report: Version 7.0.0 r153351 '7.0.0_BETA1'
00:00:13.075704 VMMDev: Guest Log: VBoxMP::DriverEntry: VBox WDDM Driver for Windows 8+ version 7.0.0r153351 rel, 64 bit; Built Aug 25 2022 19:58:37
00:00:15.941188 VMMDev: Guest Log: 19:25:12.542814 main     VBoxService 7.0.0_BETA2 r153583 (verbosity: 0) win.amd64 (Sep 12 2022 13:32:26) release log
The log messages above are from another of your log files (from the audio/microphone topic), but they show that you currently have mixture of two VirtualBox Guest Additions (GA) versions installed. The exception occurs inside the VirtualBox graphics implementation, so this could be related. I'd suggest to do the following:

Start the VM and shut it down from within the guest OS, to bring the VM into the powered-off state. In the VM configuration, disable 3D Acceleration to circumvent the crash. Start the VM, uninstall the GA, reboot the guest OS, re-install the GA, reboot the guest OS, then shut down the guest OS. Start the VM, shut it down from within the guest OS, and check the version infos in the VBox.log file for consistency. If all is well, re-enable 3D Acceleration.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Beta 2: couldn't restore the saved state.

Post by scottgus1 »

Adding in this topic from Air Force One where problems restoring from saved state are discussed too: viewtopic.php?f=6&t=107201&p=524517#p524517
Air Force One
Posts: 105
Joined: 6. Oct 2017, 16:54
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Windows
Location: Germany

Re: Beta 2: couldn't restore the saved state.

Post by Air Force One »

fth0 wrote: The log messages above are from another of your log files (from the audio/microphone topic), but they show that you currently have mixture of two VirtualBox Guest Additions (GA) versions installed. The exception occurs inside the VirtualBox graphics implementation, so this could be related.
Thank you for your response. I checked your information, and you are right, without any surprise. I have the binaries and extension pack from Beta 2, but the additions are still from Beta 1.

So, I try to install the Beta 2 additions once again, and in spite of the successful message and reboot at the end, I still have the additions from Beta 1. Posting my installation logs here, so you can see, that there are some errors here. Totally 3 drivers couldn't be installed, but installation process somehow thinks, that everything is OK and offers a reboot.

I think that the problem is certificates and the signature of the drivers. It is already posted here two times: one and two.

I also can't install those drivers manually, so I have to download the ISO from Beta 1 and return to this version of additions. Otherwise, I have no working configuration; the drivers from current release couldn't work with current beta, which I already reported previously.
Attachments
Vbox_instal_logs_25092022.7z
Installtion logs
(1.74 KiB) Downloaded 42 times
Air Force One
Posts: 105
Joined: 6. Oct 2017, 16:54
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Windows
Location: Germany

Re: Beta 2: couldn't restore the saved state.

Post by Air Force One »

scottgus1 wrote:Adding in this topic from Air Force One ...
Thank you for your replay and helpful addition. But it looks like we have some issues with current beta, which I report in this topic. And other topic is about current release, saved state and updates, applied to the host.

So, I don't know if those topics are connected or should be connected.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Beta 2: couldn't restore the saved state.

Post by fth0 »

Air Force One wrote:I think that the problem is certificates and the signature of the drivers.
Agreed. Perhaps its the same or a similar problem as with the VirtualBox test builds and development snapshot. Please try the following:

Mount the VirtualBox Guest Additions ISO in the guest, open a Command Prompt with administrator privileges, navigate to the cert folder of the virtual DVD and execute the following command, then retry the installation:

Code: Select all

VBoxCertUtil.exe add-trusted-publisher vbox*.cer --root vbox*.cer
If it works and if you can revert the changes in your guest, try each of the three certificates in the folder on its own. Start with vbox-legacy-timestamp-ca.cer.

Regarding driver installation logs, take a look at C:\Windows\INF\setupapi.dev.log, which contains the most detailed information regarding the certificate checks.
Air Force One
Posts: 105
Joined: 6. Oct 2017, 16:54
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Windows
Location: Germany

Re: Beta 2: couldn't restore the saved state.

Post by Air Force One »

fth0 wrote:Agreed. Perhaps its the same or a similar problem as with the VirtualBox test builds and development snapshot. Please try the following:
Thank you once again for your response. The idea of manual certificate installation, as in latest also not properly signed betas, is completely right! I saw it in this ticket and should think about it by myself.

Foremost, but this is probably not this important: the Beat 1 ISO has 3 certificates in cert folder; all the vbox certificates. But the Beta 2 ISO also has this 2 additional root certificates, which previously only were available in the installation packet itself. Or were present after the installation in the cert folder in binary folder of Vbox. But this probably is not important in our case.

So, after I installed the timestamp certificate manually, I could install the additions properly. The only problem here: as soon as the new version of graphic driver is installed and loaded, I see garbage on screen and a couple of seconds later the whole Vbox crashes completely. Same thing happens if I try to install the video driver manually.

The only way right now to install the Beta 2 drivers is to uninstall any previous version first and install the Beat ones after. This way I managed to install the additions from the Beta 2. I put the host and guest logs here, so if somebody would like to investigate this crash - the information is already here.
Attachments
VBox_guest_logs_install_25092022.7z
Logs from the guest
(10.23 KiB) Downloaded 39 times
VBox_host_logs_install_25092022.7z
Logs from the host
(59.28 KiB) Downloaded 40 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Beta 2: couldn't restore the saved state.

Post by fth0 »

With the certificate issue out of the way, the new crash is only indicated in the VBoxHardening.log file as 0xc0000374 (STATUS_HEAP_CORRUPTION).

The current and the older VBoxHardening.log files indicate that some "software" patched the VirtualBoxVM.exe and some major Windows DLLs in memory. The VirtualBox hardening code then patched them back to their original content. If the "software" relied on the patches to exist, this could eventually lead to any type of crash.

If this "software" has to do with the portability property of your installation, you could test on your non-portable system to rule out this possibility.
Air Force One
Posts: 105
Joined: 6. Oct 2017, 16:54
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Windows
Location: Germany

Re: Beta 2: couldn't restore the saved state.

Post by Air Force One »

fth0 wrote:With the certificate issue out of the way, the new crash is only indicated in the VBoxHardening.log file as 0xc0000374 (STATUS_HEAP_CORRUPTION).
Once again, thank you for your replay, but I have some questions.
Number one: are there any descriptions about hardening log file? What those 2 hexadecimal numbers at the beginning of every line means?

Next question: I couldn't find the STATUS_HEAP_CORRUPTION anywhere in those logs, which file and line is it?
Then: what do you mean by software, which patches something? In which lines I can see this? There is no patching happens, as far as I know.

The patching could be explainable in the case of host updates. But this case isn't about it. What I really see is an exception in gdi32.dll. And this probably causes the crash.

One more thing I forgot to mention last time: as I couldn't restore the state, I try to drop the saved state by simple resetting the VM from manager. But as soon as I start the VM, the saved state is here and VBox tries to restore with another crash. As in Beta 2 I managed to finally reset it by restarting it and saving the state at the very beginning of the guest Windows boot. This way, one gets the saved state, which could be restored. Afterwards, everything works as it should with saved states. I put my logs here as I did it last time.

I got an idea, that this crash happens because I saved the VM started from the manager and then try to start it from the command line. But this is not a case, because it also happens when you restore from the manager itself.

The crash because of some patching could happen in the host update scenario, which isn't happening here. So if hardening check happens at the start of the VM and then after host updates, when we save the VM's state, it of course possible, that some DLLs were changed through update. And therefore hardening see the changes and somehow causes the crash through revert of files or data.
Attachments
VBox_logs_save_crash_after_reset29092022.7z
Logs of reset and crash after reset
(65.22 KiB) Downloaded 29 times
VBox_logs_save_crash_29092022.7z
Logs of crash after save
(45.25 KiB) Downloaded 49 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Beta 2: couldn't restore the saved state.

Post by fth0 »

Air Force One wrote:Number one: are there any descriptions about hardening log file? What those 2 hexadecimal numbers at the beginning of every line means?
The only documentation is the VirtualBox source code, which is available at the VirtualBox download sites. Each line of the VBoxHardening.log file begins with the process ID and the thread ID of the thread creating the log message. Don't ask me why they are displayed in hex. ;)
Air Force One wrote:Next question: I couldn't find the STATUS_HEAP_CORRUPTION anywhere in those logs, which file and line is it?
VBoxHardening.log file from VBox_host_logs_install_25092022 wrote:
2988.5f4: supR3HardNtChildWaitFor[2]: Quitting: ExitCode=0xc0000374 (rcNtWait=0x0, rcNt1=0x0, rcNt2=0x103, rcNt3=0x103, 217414 ms, the end);
764.ef8: supR3HardNtChildWaitFor[1]: Quitting: ExitCode=0xc0000374 (rcNtWait=0x0, rcNt1=0x0, rcNt2=0x103, rcNt3=0x103, 218462 ms, the end);
The exit code can be found in the last log message. The Microsoft documentation can be found in 2.3.1 NTSTATUS Values.
Air Force One wrote:Then: what do you mean by software, which patches something? In which lines I can see this?
VBoxHardening.log file from VBox_host_logs_install_25092022 wrote:
764.ef8: kernel32.dll: Differences in section #2 (.rdata) between file and memory:
[...]
764.ef8:   Restored 0x2000 bytes of original file content at 00007ffe82c61000
This is one of several examples, which you can find near the beginning of the VBoxHardening.log file. The log messages indicate that parts of the contents of kernel32.dll as seen inside the memory space of the VirtualBoxVM process are different from the corresponding parts of the file on disk. This means that some software has modified those parts of kernel32.dll in memory. VirtualBox tries to revert those modifications, and only continues to run the VM if it is successful in doing that. If the other software doesn't expect that to happen, all sort of consequences can arise.

One type of software often seen modifying Windows core DLLs is so-called security software, which behaves just like malware in this respect. That's one of the reasons why neither Microsoft nor security experts like 3rd-party AV software. ;)

Regarding your theory of relating the patching to host OS updates, note that all your VBoxHardening.log files show the same modifications.

The exceptions in your latest VBox.log files took place in the VMSVGA FIFO thread, so they are most probably related to the graphics implementation.
Post Reply