VirtualBox "ESX" Implementation

Discussions related to using VirtualBox on Linux hosts.
bundabrg
Posts: 47
Joined: 22. Feb 2008, 04:01

VirtualBox "ESX" Implementation

Post by bundabrg »

Hi All, I've been busy as we've just had another baby, however I'm back and I've got something for you to mull over.

For the last few months I've been designing a bare-metal implementation of VirtualBox. It works reasonably well. By bare metal, I mean that a very small host is installed onto the system in as uncomplicated a manner as possible. Currently I've got it down to three files (and any images of course). A vmlinuz, an initrd, and a root squashfs system, as well as grub in the mbr. When the host boots up, the initrd will mount the root squashfs system and aufs (think unionfs) it with a tmpfs system.

The squashfs system basically boots a stripped down version of linux that has enough to get X running and VirtualBox. It then starts my own service that automatically scans for images in the original root, automagically creates bridges and network interfaces as required, and starts them up, optionally full screen. The host is entirely anonymous. That is, all its bridges and interfaces have no IP's. As far as the user is concerned, they don't even know there is a host os and that there was just a different starting splash screen (briefly) which actually just displays my company logo using usplash.

Currently I'm at the point where if there are three images, and they are all set to start full screen, the client will start each one on a separate X display. My client has a 'kvm-like' app where if you hit scroll lock a couple of times, it pops up a menu on the active screen and you can choose a display to change to. I of course disable the host key, the X zap key and the X vt change keys.

Ok, now for the mulling time. I would like to be able to control my service that sitting on the host FROM one of the VM's. That is, I'd like to write an app that allows me to start/stop VM's, update the hostos squashfs file, configure VM's such as change memory allocation, and upload new VM's to it. You could think of it as similar to the VirtualBox GUI, but with a few hostos specific options as well and it would also require a password to get into (which would be stored on the hostos itself). It would be communicating from the Virtual World into the Real World.

However, I can't think of HOW I can get the client to connect to a service on the hostos. Oh, I can easily write the app, and I can easily write a server component that it could connect to, but the hostos is anonymous and has no IP.

Things I've thought of so far: -
  • I could give the hostOS a specific IP, but what IP would I give it? It has no idea what IP addresses the VM's are going to use, and its supposed to be extremely generic. If I change my VM's IP, I don't want to have to go and change my hosts IP as well. A
  • I could give ALL hosts a generic ip of something like 169.254.2.3 and intercept traffic to that IP. However if the host is multihomed, and different VM's are bridged onto different network cards, you can't have the same ip address on separate bridges.
  • Perhaps send via a multicast. That is, maybe the client communicates over 255.0.0.124 (or whatever, I've never dealt with multicast addresses before). No idea of the implications, but probably similar to broadcast (see below)
  • Perhaps send via broadcasts. That is, 255.255.255.255. Not sure how this affects if there is more than one host on the real network (ie, a broadcast leaking off one host and getting replied back from several others)
  • Is there someway to communicate via the VirtualBox Guest Additions? This would probably be the best way.
Any thoughts?

The only way I can think of that is more a work-around is to run full screen on one of the displays the configuration client FROM the hostos. That is, you hit scroll-lock twice, and choose the 'Configuration' screen which displays the configuration which is acutally running ON the hostos. Unfortuantely it means a lot of the stuff I want to do (update hostos, uload images, move images from one host to another, remote access admin host) can't easily be done.

- Brendan
bundabrg
Posts: 47
Joined: 22. Feb 2008, 04:01

Post by bundabrg »

Oh, and before someone says "Thats not bare-metal", its true, but only a very little bit of work from real bare-metal. Here's my idea for the next step.

Picture this. You have a rack with 5 blade servers. These servers have NO disks, and just provide the CPU.

They boot via PXE, and pull the vmlinuz and initrd from a "Master Server". The initrd determines that it was started via PXE rather than local boot, so it connects BACK to the "Master Server" and using readonly nbd mounts the root squashfs and aufs (union) it with tmpfs. Anyone familiar with LTSP would know that it uses a similar method. Very easy to implement.

It would then connect to an image server (perhaps the same server as the "Master Server") and connect to one or more images. Either via NBD or via some other method (I'd prefer writable raw network blocks to something like NFS peronsally but thats because I think NFS is just an absolute pain AND locks you a bit to *nix world). It would start these images and the KVM like app allows switching between them. By KVM I mean 'Keyboard Video Mouse' and not 'Kernel Virtual Machine' (Anyone ever tried to find KVM's home page on Google??)

On any machine (lets say your notebook), one can run the "Management Tool" that allows one to configure what VM's run on which machine (it connects to the Master Server and updates configuration there), clone them between them, and migrate VM's. You can stop and start VM's as well. Perhaps it would allow connection to the VRDP port, likely by tunneling the RDP through itself so you don't have to have VRDP listening on public ports. The management tool is OS agnostic, so I'd likely write it in wxPython. The KVM tool is in Python as well, and uses xlib for keygrabbing.

In this scenario the "Master Server" can provide IP's for the hosts on the blades so its easy. However I'd also like a 'smaller business' approach where its all done via a local boot and the Management Tool can be run either from another machine OR from within one of the virtual clients.

I'd also like to have a scenario where one can VM a normal workstation (lets say an XP box), such that it boots the hostos, then boots the XP image full screen. The user just uses the machine as normal. Behind the scenes the image is backed up, and if the machines ever dies it can be moved to another machine easily with no reconfiguring. Indeed you can keep a copy of the machine locally (assuming the physical machine is located on the otherside of the world) and test stuff on the local copy before pushing to the remote. You'll get a speed loss, mainly because you don't have SMP etc, but on certain "special" workstations this is fine.

- Brendan
Phobos
Volunteer
Posts: 117
Joined: 28. Feb 2008, 00:15

Post by Phobos »

bundabrg wrote:Oh, and before someone says "Thats not bare-metal", its true, but only a very little bit of work from real bare-metal. Here's my idea for the next step.
Well... If KVM is bare-metal, then this is bare metal too heh...

Have you checked how KVM accomplishes this?
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Post by Technologov »

To make VirtualBox more "bare-metal" type, it needs better infrastructure. And mini-distro, of course.

One possible direction to look at is Red Hat's oVirt, which is mini distro + management layer + web GUI.

It is based on libvirt, a virtualization management abstration layer, that can theoretically support multiple virtualization engines.

Both libvirt and oVirt need modifications to work with VBox.

My View:
Of course VBox won't suddenly become type 1 hypervisor, but honestly, I don't see any advantage of type 1 systems (so-called "bare-metal"), compared to type 2. (Hosted virtualizers).
I prefer type 2 systems in all scenarios I can think of.

Type 1 hypervisors are more difficult to write, maintain & test, and they provide no advantages.

Another topic:
Innotek were designing their own type 1 hypervisor, called "Hyperkernel".
But with Sun's aquisition, I don't know what will come out of it. Sun already have Sun xVM Server.

Is there any info on what happened with Innotek's Hyperkernel ?

-Technologov
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: VirtualBox+Oracle ExtPack
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

Brendan, this is just a placeholder response because I would like to come back on this one tomorrow. I had to use the search-word "baby" to find this post again :lol: Congrats to you and your partner, BTW. You might be interested in this topic as a point of comparison: HOWTO: Diskless Windows guest on diskless Linux host and for other readers a cross-reference to another post that we've both contributed to: Virtualbox and Terminal Server -> how to solve?.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
pennymarkt
Posts: 5
Joined: 1. Oct 2008, 17:04

Post by pennymarkt »

@bundabrg - very interesting project - plz keep us updated! :)
TerryE wrote:Brendan, this is just a placeholder response because I would like to come back on this one tomorrow. I had to use the search-word "baby" to find this post again :lol: Congrats to you and your partner, BTW. You might be interested in this topic as a point of comparison: HOWTO: Diskless Windows guest on diskless Linux host and for other readers a cross-reference to another post that we've both contributed to: Virtualbox and Terminal Server -> how to solve?.
SWIM looks awesome... will check it this upcoming weekend... - any idea if it is possible to not use the pxe-feature... - i thougt of an ultra-fast USB-stick which should be turned into a bootable one... plug/boot/play :>
myxiplx
Posts: 50
Joined: 27. Aug 2008, 07:47

Post by myxiplx »

Might be worth having a look at Sun's vXM Server. Early access versions should be around in the next few weeks and it's looking pretty nice:
http://www.sun.com/software/products/xvmserver/

From what I've read so far, the basic version is going to be freely available.
Phobos
Volunteer
Posts: 117
Joined: 28. Feb 2008, 00:15

Post by Phobos »

myxiplx wrote:Might be worth having a look at Sun's vXM Server. Early access versions should be around in the next few weeks and it's looking pretty nice:
http://www.sun.com/software/products/xvmserver/

From what I've read so far, the basic version is going to be freely available.
Sorry, but I think it's not really relevant to Brendan's project....

Either way, the "basic version", xVM hypervisor is already available as open source in opensolaris... the rest of xVM Server will also be open source (including the BUI and management bits), with an early access open source code already available...

Brendan's project is built around VBox, with no relation to xVM Server or Xen for that matter... while being more structurally similar to RedHat's KVM...

up until now, I had though that oVirt was only the web interface for managing guests... but having read that it hosts VMs by itself, I agree with Technologov that it could be a better reference for this project

... and I also share the question about what happened to innotek's type-1 hyperkernel... I had been waiting for it since it appeared (almost hidden) on innotek's site and has since disappeared with no traces to be found...
NeBlackCat
Posts: 154
Joined: 3. Aug 2008, 13:37

Post by NeBlackCat »

IIRC (this is from years ago, so things may have changed) VMWare Workstation has a backdoor API that your client app can use to communicate with your host app without requiring or using networking.

I think it was based on a software interrupt. That is, your client app stuffs some things in registers and executes an INT NN instruction, and VMware traps this and somehow reflects it through to the host side for handling. And it presumably works in reverse too.

I don't know if VB has anything similar but, if not, it should do!

Are you documenting how you go about making your mini distro, and your self developed tools? I'd be interested in helping out, as I was going to do the exact same thing myself. I'm much more of a Windows than Linux person though.
vkov_tinsky
Volunteer
Posts: 218
Joined: 5. Apr 2008, 20:18

Post by vkov_tinsky »

NeBlackCat wrote:I don't know if VB has anything similar but, if not, it should do!
This sounds like what you're talking about.

Regards,
VT
NeBlackCat
Posts: 154
Joined: 3. Aug 2008, 13:37

Post by NeBlackCat »

Indeed - thanks! It doesn't seem to documented anywhere, other than the source code, though. That's a shame.

Cheers
NBC
grantmasterflash
Posts: 36
Joined: 3. Apr 2008, 03:58

Post by grantmasterflash »

So all you're wanting to really do is have an application on one of the guests communicate with the host? I'm not sure I'm understanding the complexity here. Where do the guests get their IPs (or do they?). If they're all on one network then just bridge the host physical network device to the bridge used by the hosts and assign a static IP to it. This seems so easy that I'm sure there's a reason you're not doing it considering the amount of work you've put into this so far.

Also I wouldn't mind getting your kernel, initrd and rootfs.gz. I'm building the same sort of system but with only one Full screen VM and a couple that are behind the scenes. As in your case the host OS needs to be as small as possible so rebooting the kiosk looks like you're rebooting the Guest. I have the host sitting on a bridge and the guests also sitting on the same bridge. The host gets it's IP from a dhcp server residing on one of the guests (As do the other guests).
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: VirtualBox+Oracle ExtPack
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

Brendan, you might be interested in my topic HOWTO: Use Guest Properties as a way of having multiple guests use the same immutable system image but still be able to personalise each through guest properties.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
NeBlackCat
Posts: 154
Joined: 3. Aug 2008, 13:37

Post by NeBlackCat »

grantmasterflash wrote:So all you're wanting to really do is have an application on one of the guests communicate with the host? I'm not sure I'm understanding the complexity here. Where do the guests get their IPs (or do they?). If they're all on one network then just bridge the host physical network device to the bridge used by the hosts and assign a static IP to it. This seems so easy that I'm sure there's a reason you're not doing it considering the amount of work you've put into this so far.
Yes, but to do it in a way that does not require networking, or any specific network configuration, to be set up on either guests or host. That would be impossible with physical machines, but in a virtualised environment there are typically hooks provided in the virtualisation software for this communication to occur.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: VirtualBox+Oracle ExtPack
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

NeBlackCat, why don't you just read the topic that I referred to in my previous post? I have been using guest properties to do specifically this.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
Post Reply