My company distributes a local web application that runs on an Ubuntu VM. For our VBox deployment, we normally require that clients have a physical server as the host.
One of our clients has their whole network running on Azure and their IT opted to try installing our VM inside of their Azure Windows Server VM. The nested Ubuntu VM showed some performance issues, including a very slow boot up and high CPU usage on the host. Most of the time it runs well enough for them to use our application, but sometimes the Ubuntu VM is slow to the point of being unusable and takes up almost 100% of the host CPU. A reboot of their Window Server VM normally resolves this but sometimes it does not.
I know nested virtualization is not supported by VirtualBox or recommended, but I'm wondering if anyone has tried running VirtualBox inside of an Azure VM. I'd like to know if there's anything we can do to mitigate these performance issues and make the Ubuntu VM run more reliably?
The client doesn't want to fork up any costs to accommodate one of our standard deployments, but they're also frustrated by these unpredictable issues. So we're in a bit of a tight spot.
I've attached the log files. The hardening log was too big so I've split it up. If anyone has suggestions those would be much appreciated!
The Windows Server VM is running an Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz with 2 cores. We've tried running the Ubuntu VM with 1 and 2 cores - neither option seems to improve the performance when it's running slowly.
Ubuntu 18.04 VM nested in Windows Server Azure VM
Ubuntu 18.04 VM nested in Windows Server Azure VM
- Attachments
-
- VBoxHardening - 2.log
- (122.45 KiB) Downloaded 49 times
-
- VBoxHardening - 1.log
- (121.93 KiB) Downloaded 49 times
-
- VBox.log
- (125.67 KiB) Downloaded 46 times
Last edited by fstephane on 6. May 2021, 23:17, edited 2 times in total.
Re: Ubuntu 18.04 VM nested in Windows Server Azure VM
I ran into the attachment limit in the original post - here's the last part of the hardening log:
- Attachments
-
- VBoxHardening - 3.log
- (127.22 KiB) Downloaded 43 times
-
- Site Moderator
- Posts: 20945
- Joined: 30. Dec 2009, 20:14
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Windows, Linux
Re: Ubuntu 18.04 VM nested in Windows Server Azure VM
Logs fit in one file when zipped.
Hardening exit code = 0, no hardening issues.
Host processor capability is too little for this VM:
Hardening exit code = 0, no hardening issues.
Host processor capability is too little for this VM:
Even if more processors are put into the intermediary Server 2016 VM, the capabilities needed for smoother nested virtualization are not present (all four need to be (1)'s )Host CPU status:
00:00:12.337166 CPUM: Logical host processors: 2 present, 2 max, 2 online, online mask: 0000000000000003
00:00:12.337168 CPUM: Physical host cores: 1
00:00:12.337939 Full Name: "Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz"
Guest CPU settings:
00:00:11.439345 NumCPUs <integer> = 0x0000000000000002 (2)
Intel Nested Virtualization status: guest (host) (1) is enabled
00:00:12.337959 VMX - Virtual-Machine Extensions = 0 (1)
00:00:12.337980 Ept - Extended Page Tables = 0 (1)
00:00:12.337983 UnrestrictedGuest - Unrestricted guest = 0 (0)
00:00:12.337986 VmcsShadowing - VMCS shadowing = 0 (0)
Actually it is, if Virtualbox is on both layers, and if the CPU is properly capable. Neither is the case here.fstephane wrote:nested virtualization is not supported by VirtualBox
'Tis a Scylla vs Charybdis moment. Either your customer's going to have to eat it or your company is. I'd try putting the Ubuntu VM directly into Azure, which could be possible: https://azure.microsoft.com/en-us/overv ... arketplace.fstephane wrote:The client doesn't want to fork up any costs to accommodate one of our standard deployments, but they're also frustrated by these unpredictable issues.
Re: Ubuntu 18.04 VM nested in Windows Server Azure VM
Thanks for your response, this is really helpful!
Concerning those CPU vt-x features, I'm confused as to which of them are actually active.
Concerning those CPU vt-x features, I'm confused as to which of them are actually active.
The VMX and EPT features have a "0 (1)" value. Does that mean they are active on the processor or inactive?Intel Nested Virtualization status: guest (host) (1) is enabled
00:00:12.337959 VMX - Virtual-Machine Extensions = 0 (1)
00:00:12.337980 Ept - Extended Page Tables = 0 (1)
00:00:12.337983 UnrestrictedGuest - Unrestricted guest = 0 (0)
00:00:12.337986 VmcsShadowing - VMCS shadowing = 0 (0)
-
- Site Moderator
- Posts: 20945
- Joined: 30. Dec 2009, 20:14
- Primary OS: MS Windows 10
- VBox Version: PUEL
- Guest OSses: Windows, Linux
Re: Ubuntu 18.04 VM nested in Windows Server Azure VM
If I understand correctly, the (#) in parentheses shows what Virtualbox is receiving, and the number before the ( shows what Virtualbox is sending into the VM.
So, VMX and Ept are on, wheres UnrestrictedGuest and VmcsShadowing are off. And especially the last one "VmcsShadowing" is critical for decent nested virtualization.
This list of important CPU features was compiled by our forum guru and hex whisperer 'fth0'.
So, VMX and Ept are on, wheres UnrestrictedGuest and VmcsShadowing are off. And especially the last one "VmcsShadowing" is critical for decent nested virtualization.
This list of important CPU features was compiled by our forum guru and hex whisperer 'fth0'.
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Ubuntu 18.04 VM nested in Windows Server Azure VM
Microsoft Azure is based on Hyper-V, and Hyper-V officially supports nested virtualization only for Hyper-V inside Hyper-V, according to Run Hyper-V in a Virtual Machine with Nested Virtualization. When using Azure, you're really dealing with double nested virtualization, since the Azure VM is itself already running in a nested Hyper-V:
Regarding the CPU features, VMX and EPT are provided to the Azure VM1 (VirtualBox host), so that VirtualBox can run VM2. VMCS Shadowing is critical for performance, but if it is used in the setup at hand can not be derived from the VirtualBox log messages (*), so it may or may not be the key here.
(*) In a typical VirtualBox nested virtualization setup, we have PM0 > HV1 > VM1 > HV2 > VM2, with HV1 and HV2 being the VirtualBox hypervisors. Then we can look into the VBox.log file of HV1 to see if the physical CPU (PM0) provides the VMCS Shadowing CPU feature, which enables HV1 to efficiently handle the VMCS structures created by HV2.
Code: Select all
PM0 > HV0 > VM0 > HV1 > VM1 > HV2 > VM2:
PM0 is the Azure hardware
HV0 is the Hyper-V hypervisor running on PM0
VM0 is the Azure Root OS running in HV0
HV1 is the Hyper-V hypervisor running next to VM0
VM1 is an Azure VM running in HV1
HV2 is a VirtualBox hypervisor running in VM1
VM2 is a VirtualBox VM running in HV2
(*) In a typical VirtualBox nested virtualization setup, we have PM0 > HV1 > VM1 > HV2 > VM2, with HV1 and HV2 being the VirtualBox hypervisors. Then we can look into the VBox.log file of HV1 to see if the physical CPU (PM0) provides the VMCS Shadowing CPU feature, which enables HV1 to efficiently handle the VMCS structures created by HV2.