Application popup: VirtualBoxVM.exe - Application Error : The exception Integer division by zero.

[Syntetik Vision]

Hi all

I have a HP server with Windows Server 2016 Standard fully patched and update, running a lot of Centos 7 (fully patched and update).

In the last month, the Virtualbox start to give random errors crashing the vm's (there is no rule or logic in this crashs).

Windows Event Viewer write an entry each time the vm crash with this error:

Application popup: VirtualBoxVM.exe - Application Error : The exception Integer division by zero.
(0xc0000094) occurred in the application at location 0x00007FFC0F96D60B.

Click on OK to terminate the program

I've read that 3D acceleration could lead to this type of errors, but the guest os's dont have gui. Anyway, i removed the 3d acceleration from all the vms, but the problem still happen.

Already tried with different guest kernels, but no luck.

Windows Server 2016 Standard

OS Name: Microsoft Windows Server 2016 Standard
OS Version: 10.0.14393 N/A Build 14393

Centos 7

Version: centos-release-7-8.2003.0.el7.centos.x86_64
Kernel: 3.10.0-1127.10.1.el7.x86_64

Virtualbox 6

VirtualBox Graphical User Interface
Version 6.0.20 r137117 (Qt5.6.2)

Thank you in advance

[scottgus1]

Please pick one guest. Start the guest from full normal shutdown, not save-state. Run until you see the problem happen, then shut down the guest from within the guest OS if possible. If not possible, close the Virtualbox window for the guest with the Power Off option set.

Right-click the guest in the main Virtualbox window's guest list, choose Show Log. Save the far left tab's log, zip it, and post the zip file, using the forum's Upload Attachment tab.

[Syntetik Vision]

Hi Scoot,

Thank you for your words. Unfortunately, i can't shutdown from guest OS or from the guest window. When the crash happens, a dialog box popup and i need to click on OK.

After this, the machine will go to Aborted state, and then i start it again manualy. Next time a guest goes down, i will get the log and upload it.

Best regards,

[scottgus1]

i can't shutdown from guest OS or from the guest window. When the crash happens, a dialog box popup and i need to click on OK.

That's enough. After this happens, please zip and post the vbox.log.

[Syntetik Vision]

Log from one guest, on this morning crash.

If you need anything else, please let me know.

Thank you.

[scottgus1]

17:24:16.724516 !!! rtR3WinUnhandledXcptFilter caught an exception on thread 00000000000016f8!!!
17:24:16.724615 ExceptionCode=0xc0000094 ExceptionFlags=0x00000000 ExceptionAddress=00007ffc0f96d60b
......
17:24:16.726065 Xcpt PC : 00007ffc0f96d60b - 0x000bd60b bytes into C:\Program Files\Oracle\VirtualBox\VBoxDD.DLL

So the guest died in VboxDD.dll. Exactly what that dll does, I don't unfortunately know.

You can inspect other crashed logs by looking for the line:

!!! rtR3WinUnhandledXcptFilter caught an exception....

Then look for the "ExceptionAddress=xxxxxxxxxx" two lines down. Copy and search for the hex address numbers, it will go to the line starting with "Xcpt PC", which will show what dll or exe crashed. (This process taught to the forum by guru 'fth0')

If they're all Virtualbox dlls, try a reinstall of Virtualbox using right-click-Run-As-Administrator.

Also, if these guests are pure data crunching and not GUI needing audio and video, consider disabling the audio controller in each guest.

[Syntetik Vision]

Thank you so much for the help and tips

The weirdest thing about this error, is that the guest's were fine for about 5 months and then start with this crashs.

I've removed 3d aceleration from all, update the guest os for the last kernel and updates, and the Guest additions too, but no luck.

I have another host running guests like this, and never had this crash. But the hardware is completely different, but in this host i never update the virtualbox.

They have different versions of VirtualBox, maybe this can explain some issues, don't know.

Best regards

[Syntetik Vision]

Searching for the thread on the same log, found another clue that may point to network interface.

17:24:16.725524 **************** **** <ditto x 35>
17:24:16.725524 0000000009b8fff0 13f0: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
17:24:16.725545 Thread ID: 00000000000016f8
17:24:16.725551 Thread name: INTNET-XMIT
17:24:16.725553 Thread IPRT: 0000000004a90b40

Don't know what INTNET-XMIT, but maybe is something related with network interface. What you think about this ?

Thank you in advance

[fth0]

So the guest died in VboxDD.dll. Exactly what that dll does, I don't unfortunately know.

VBoxDD.dll contains the majority of the implementation of all virtual devices, so to know that is not sufficient on its own.

Don't know what INTNET-XMIT, but maybe is something related with network interface. What you think about this ?

Good guess.

00:00:02.175572   Network              <string>  = "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #113" (cb=71)
00:00:02.175573   Trunk                <string>  = "\DEVICE\{48F54713-8E73-4182-B547-AD60092555EE}" (cb=47)
00:00:02.175597   Network              <string>  = "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #85" (cb=70)
00:00:02.175598   Trunk                <string>  = "\DEVICE\{A2DC1010-DA5A-48F2-9CFF-CB39ECE8587A}" (cb=47)
00:00:02.175621   Network              <string>  = "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #112" (cb=71)
00:00:02.175622   Trunk                <string>  = "\DEVICE\{B5FC9655-C1C5-41EC-A869-66B6BA093DFA}" (cb=47)
00:00:02.175645   Network              <string>  = "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device" (cb=66)
00:00:02.175646   Trunk                <string>  = "\DEVICE\{FE61EEC2-5E0A-4D66-B234-504950565706}" (cb=47)
[...]
00:00:02.433042 E1000#0: Chip=82540EM LinkUpDelay=3000ms EthernetCRC=on GSO=enabled Itr=disabled ItrRx=enabled TID=disabled R0=enabled GC=enabled
00:00:02.433155 IntNet#0: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #113} enmTrunkType=3 szTrunk={\DEVICE\{48F54713-8E73-4182-B547-AD60092555EE}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false
00:00:02.433652 E1000#1: Chip=82540EM LinkUpDelay=3000ms EthernetCRC=on GSO=enabled Itr=disabled ItrRx=enabled TID=disabled R0=enabled GC=enabled
00:00:02.433757 IntNet#1: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #85} enmTrunkType=3 szTrunk={\DEVICE\{A2DC1010-DA5A-48F2-9CFF-CB39ECE8587A}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false
00:00:02.434211 E1000#2: Chip=82540EM LinkUpDelay=3000ms EthernetCRC=on GSO=enabled Itr=disabled ItrRx=enabled TID=disabled R0=enabled GC=enabled
00:00:02.434315 IntNet#2: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #112} enmTrunkType=3 szTrunk={\DEVICE\{B5FC9655-C1C5-41EC-A869-66B6BA093DFA}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false
00:00:02.434845 E1000#3: Chip=82540EM LinkUpDelay=3000ms EthernetCRC=on GSO=enabled Itr=disabled ItrRx=enabled TID=disabled R0=enabled GC=enabled
00:00:02.434955 IntNet#3: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device} enmTrunkType=3 szTrunk={\DEVICE\{FE61EEC2-5E0A-4D66-B234-504950565706}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false

When the VM is started, there are 4 network adapters configured, created and connected to host network interfaces. When the VM is suspended about 6 hours later, a fifth host network interface appears:

06:16:58.472384 IntNet#4: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #111} enmTrunkType=3 szTrunk={\DEVICE\{B6672E97-7C83-4163-965B-34DD0C17E4E5}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false

What can you tell us about the host network interfaces with the interesting names and the host network configuration?

[Martin]

RNDIS is typcally an USB connected network devices like a tethered mobile phone or similar.

[Syntetik Vision]

The guest has 4 network interfaces connected in Bridged Adapter mode, each one is connected to RNDIS device (image with connected devices below)

The guest cant have more than 4 network interfaces, so i can't understand how and why there is a fifth network interface

The question is why this is crashing randomly, after months without a issue, without any logic or pattern ...

Thank you all

[Syntetik Vision]

RNDIS is typcally an USB connected network devices like a tethered mobile phone or similar.

This devices are 4G Mobile Hotspot routers from Alcatel.

[Syntetik Vision]

So the guest died in VboxDD.dll. Exactly what that dll does, I don't unfortunately know.

VBoxDD.dll contains the majority of the implementation of all virtual devices, so to know that is not sufficient on its own.

Don't know what INTNET-XMIT, but maybe is something related with network interface. What you think about this ?

Good guess.

00:00:02.175572   Network              <string>  = "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #113" (cb=71)
00:00:02.175573   Trunk                <string>  = "\DEVICE\{48F54713-8E73-4182-B547-AD60092555EE}" (cb=47)
00:00:02.175597   Network              <string>  = "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #85" (cb=70)
00:00:02.175598   Trunk                <string>  = "\DEVICE\{A2DC1010-DA5A-48F2-9CFF-CB39ECE8587A}" (cb=47)
00:00:02.175621   Network              <string>  = "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #112" (cb=71)
00:00:02.175622   Trunk                <string>  = "\DEVICE\{B5FC9655-C1C5-41EC-A869-66B6BA093DFA}" (cb=47)
00:00:02.175645   Network              <string>  = "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device" (cb=66)
00:00:02.175646   Trunk                <string>  = "\DEVICE\{FE61EEC2-5E0A-4D66-B234-504950565706}" (cb=47)
[...]
00:00:02.433042 E1000#0: Chip=82540EM LinkUpDelay=3000ms EthernetCRC=on GSO=enabled Itr=disabled ItrRx=enabled TID=disabled R0=enabled GC=enabled
00:00:02.433155 IntNet#0: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #113} enmTrunkType=3 szTrunk={\DEVICE\{48F54713-8E73-4182-B547-AD60092555EE}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false
00:00:02.433652 E1000#1: Chip=82540EM LinkUpDelay=3000ms EthernetCRC=on GSO=enabled Itr=disabled ItrRx=enabled TID=disabled R0=enabled GC=enabled
00:00:02.433757 IntNet#1: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #85} enmTrunkType=3 szTrunk={\DEVICE\{A2DC1010-DA5A-48F2-9CFF-CB39ECE8587A}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false
00:00:02.434211 E1000#2: Chip=82540EM LinkUpDelay=3000ms EthernetCRC=on GSO=enabled Itr=disabled ItrRx=enabled TID=disabled R0=enabled GC=enabled
00:00:02.434315 IntNet#2: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #112} enmTrunkType=3 szTrunk={\DEVICE\{B5FC9655-C1C5-41EC-A869-66B6BA093DFA}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false
00:00:02.434845 E1000#3: Chip=82540EM LinkUpDelay=3000ms EthernetCRC=on GSO=enabled Itr=disabled ItrRx=enabled TID=disabled R0=enabled GC=enabled
00:00:02.434955 IntNet#3: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device} enmTrunkType=3 szTrunk={\DEVICE\{FE61EEC2-5E0A-4D66-B234-504950565706}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false

When the VM is started, there are 4 network adapters configured, created and connected to host network interfaces. When the VM is suspended about 6 hours later, a fifth host network interface appears:

06:16:58.472384 IntNet#4: szNetwork={HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #111} enmTrunkType=3 szTrunk={\DEVICE\{B6672E97-7C83-4163-965B-34DD0C17E4E5}} fFlags=0x8000 cbRecv=325632 cbSend=196608 fIgnoreConnectFailure=false

What can you tell us about the host network interfaces with the interesting names and the host network configuration?

After looking to the log again, and the time frame, this is not a fifth interface ... the explanation is simple, when the machine started, one of the interfaces was wrong, and then i change it from "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device" to "HostInterfaceNetworking-Remote NDIS based Internet Sharing Device #111" later ... this explains why we see 5 interfaces on the log, but as you know, Virtualbox only supports 4 network interfaces on each guest.

Thank you.

[scottgus1]

The guest cant have more than 4 network interfaces

Virtualbox only supports 4 network interfaces

https://www.virtualbox.org/manual/ch06.html#nichardware : Oracle VM VirtualBox provides up to eight virtual PCI Ethernet cards for each virtual machine.
https://www.virtualbox.org/manual/ch03. ... gs-network: The first four virtual network cards can be configured in detail in the VirtualBox Manager window. Additional network cards can be configured using the VBoxManage command.

The above Manual quote applies to the PIIX3 chipset. ICH9 allows 36 network cards, again only the first four configurable via the GUI, the rest by vboxmanage command line.

Please reconsider your thought that the 5th network card is really one of the first four, because there can be a fifth.

Four guest network cards bridged to the same location is definitely going to confuse the guest OS. Adding a fifth (and there is a fifth, Intnet# starting from 0 to 4, 0 is the first, 4 is the fifth, is going to confuse things even more. You'll need to find out what is making these connections and shut it down.

EDIT: apparently they're not to the same location, there are different numbers for the first 4: 85, 111, 112, 113. The fifth connection is to 111 again. Assuming the different numbering really means to different network locations, then it could be fine to have four different Bridged cards in the guest. But the fifth card does connect to one of the others' same destination, 111, which would be wrong. See if you can track that down.

Of course the above takes into consideration that I do not know what you are doing with this guest. Feel free to disregard everything from "Four guest network cards bridged" down, if I have missed the point of your usage scheme. From "there can be a fifth" up is fact.

[Syntetik Vision]

Thansk Scott ... i didn't know it was possible to use more than four network interfaces if you use vboxmanage, but i'm 100% sure that each guest only has 4 interfaces at the moment, since i never add another interface from command line tool.

The fifth interface that is on the logs, could be a manual change in the guest configuration, but always in the gui mode.

The links you provide have some interesting information, about the different types of network interfaces. How can i change them ?

Because the interface adapter should use Intel PRO/1000 MT Server (82545EM) since this is a server instead of Intel PRO/1000 MT Desktop (82540EM), but this is the only option in the gui configuration mode.

My virtualization skills are more for VMware/Citrix/Hyper-V/KVM, always use Virtualbox in lab test, but now i'm using it on production servers in a new project.

How can i change this for testing ? Only with VboxManage ? I don't know if this could solve the problem (think not) but i can try to see the results.

Best regards