Page 1 of 1

Ubuntu 18.04 VM nested in Windows Server Azure VM

Posted: 6. May 2021, 23:13
by fstephane
My company distributes a local web application that runs on an Ubuntu VM. For our VBox deployment, we normally require that clients have a physical server as the host.

One of our clients has their whole network running on Azure and their IT opted to try installing our VM inside of their Azure Windows Server VM. The nested Ubuntu VM showed some performance issues, including a very slow boot up and high CPU usage on the host. Most of the time it runs well enough for them to use our application, but sometimes the Ubuntu VM is slow to the point of being unusable and takes up almost 100% of the host CPU. A reboot of their Window Server VM normally resolves this but sometimes it does not.

I know nested virtualization is not supported by VirtualBox or recommended, but I'm wondering if anyone has tried running VirtualBox inside of an Azure VM. I'd like to know if there's anything we can do to mitigate these performance issues and make the Ubuntu VM run more reliably?

The client doesn't want to fork up any costs to accommodate one of our standard deployments, but they're also frustrated by these unpredictable issues. So we're in a bit of a tight spot.

I've attached the log files. The hardening log was too big so I've split it up. If anyone has suggestions those would be much appreciated!

The Windows Server VM is running an Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz with 2 cores. We've tried running the Ubuntu VM with 1 and 2 cores - neither option seems to improve the performance when it's running slowly.

Re: Ubuntu 18.04 VM nested in Windows Server Azure VM

Posted: 6. May 2021, 23:14
by fstephane
I ran into the attachment limit in the original post - here's the last part of the hardening log:

Re: Ubuntu 18.04 VM nested in Windows Server Azure VM

Posted: 7. May 2021, 00:58
by scottgus1
Logs fit in one file when zipped.

Hardening exit code = 0, no hardening issues.

Host processor capability is too little for this VM:
Host CPU status:
00:00:12.337166 CPUM: Logical host processors: 2 present, 2 max, 2 online, online mask: 0000000000000003
00:00:12.337168 CPUM: Physical host cores: 1
00:00:12.337939 Full Name: "Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz"
Guest CPU settings:
00:00:11.439345 NumCPUs <integer> = 0x0000000000000002 (2)
Even if more processors are put into the intermediary Server 2016 VM, the capabilities needed for smoother nested virtualization are not present (all four need to be (1)'s )
Intel Nested Virtualization status: guest (host) (1) is enabled
00:00:12.337959 VMX - Virtual-Machine Extensions = 0 (1)
00:00:12.337980 Ept - Extended Page Tables = 0 (1)
00:00:12.337983 UnrestrictedGuest - Unrestricted guest = 0 (0)
00:00:12.337986 VmcsShadowing - VMCS shadowing = 0 (0)
fstephane wrote:nested virtualization is not supported by VirtualBox
Actually it is, if Virtualbox is on both layers, and if the CPU is properly capable. Neither is the case here.
fstephane wrote:The client doesn't want to fork up any costs to accommodate one of our standard deployments, but they're also frustrated by these unpredictable issues.
'Tis a Scylla vs Charybdis moment. Either your customer's going to have to eat it or your company is. :lol: I'd try putting the Ubuntu VM directly into Azure, which could be possible: https://azure.microsoft.com/en-us/overv ... arketplace.

Re: Ubuntu 18.04 VM nested in Windows Server Azure VM

Posted: 7. May 2021, 18:21
by fstephane
Thanks for your response, this is really helpful!

Concerning those CPU vt-x features, I'm confused as to which of them are actually active.
Intel Nested Virtualization status: guest (host) (1) is enabled
00:00:12.337959 VMX - Virtual-Machine Extensions = 0 (1)
00:00:12.337980 Ept - Extended Page Tables = 0 (1)
00:00:12.337983 UnrestrictedGuest - Unrestricted guest = 0 (0)
00:00:12.337986 VmcsShadowing - VMCS shadowing = 0 (0)
The VMX and EPT features have a "0 (1)" value. Does that mean they are active on the processor or inactive?

Re: Ubuntu 18.04 VM nested in Windows Server Azure VM

Posted: 7. May 2021, 18:44
by scottgus1
If I understand correctly, the (#) in parentheses shows what Virtualbox is receiving, and the number before the ( shows what Virtualbox is sending into the VM.

So, VMX and Ept are on, wheres UnrestrictedGuest and VmcsShadowing are off. And especially the last one "VmcsShadowing" is critical for decent nested virtualization.

This list of important CPU features was compiled by our forum guru and hex whisperer 'fth0'.

Re: Ubuntu 18.04 VM nested in Windows Server Azure VM

Posted: 7. May 2021, 22:45
by fth0
Microsoft Azure is based on Hyper-V, and Hyper-V officially supports nested virtualization only for Hyper-V inside Hyper-V, according to Run Hyper-V in a Virtual Machine with Nested Virtualization. When using Azure, you're really dealing with double nested virtualization, since the Azure VM is itself already running in a nested Hyper-V:

Code: Select all

PM0 > HV0 > VM0 > HV1 > VM1 > HV2 > VM2:
  PM0 is the Azure hardware
  HV0 is the Hyper-V hypervisor  running on PM0
  VM0 is the Azure Root OS       running in HV0
  HV1 is the Hyper-V hypervisor  running next to VM0
  VM1 is an Azure VM             running in HV1
  HV2 is a VirtualBox hypervisor running in VM1
  VM2 is a VirtualBox VM         running in HV2
Regarding the CPU features, VMX and EPT are provided to the Azure VM1 (VirtualBox host), so that VirtualBox can run VM2. VMCS Shadowing is critical for performance, but if it is used in the setup at hand can not be derived from the VirtualBox log messages (*), so it may or may not be the key here.

(*) In a typical VirtualBox nested virtualization setup, we have PM0 > HV1 > VM1 > HV2 > VM2, with HV1 and HV2 being the VirtualBox hypervisors. Then we can look into the VBox.log file of HV1 to see if the physical CPU (PM0) provides the VMCS Shadowing CPU feature, which enables HV1 to efficiently handle the VMCS structures created by HV2.