[Free as in beer] SMF service for VirtualBox VM's

Discussions related to using VirtualBox on Solaris hosts.
dsisevran
Posts: 2
Joined: 4. Nov 2011, 12:20
Primary OS: Solaris
VBox Version: PUEL
Guest OSses: Windows 2003, FreeBSD 8

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by dsisevran »

Hi everybody!

Very usefull topic! Thanks a lot!
But what about when shutting down the host? Did the VM gracefully shutdown or did the SMF make a simple poweroff?
And how to launch the display of a VM when we a loggin on the graphical interface of the host?
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

dsisevran wrote:Hi everybody!

Very usefull topic! Thanks a lot!
But what about when shutting down the host? Did the VM gracefully shutdown or did the SMF make a simple poweroff?
You are welcome! ;)

For the VM instance (or service in general) you can configure the SMF property "vm/stop_method" with value of "savestate", which is the default in stock XML manifest. Then during the SMF service shutdown, the VM should suspend-to-disk. On subsequent SMF service startup it should resume. There are some possible conditions that would lead to incomplete or invalid saved states (i.e. restarting on another CPU hardware, too little time to save everything before the power is cut, out of free disk space, etc.) - in these cases the saved state is ignored by VirtualBox software and the VM is booted from its virtual disk (as if it hung and was reset).
To debug your VM startups/shutdowns you can see the SMF log files: /var/svc/log/site-xvm-vbox:<VM_NAME>.log
The service method script logs its work there; if a savestate atempt was untimely aborted by SMF or some other causes, this log file would like have some clues (i.e. an error after SMF stop method, or an incomplete log without the final status - if the power was cut).

Also there may be some watchdog software in your virtual OS that would detect wallclock-time jumping and would trigger a reboot.

When the host is being shut down, there are several possible SMF interactions.

First of all, all SMF services are being shut down in their dependency order, and many of them can be shut down in parallel if they do not inter-depend on each other.
If the OS is shutting down, there may be several timeouts enforced.

One is the timeout configured for the SMF service stop method (SMF property "stop/timeout_seconds"); it is 120 seconds in the default XML manifest. You can increase this timeout for the service in general or per-instance, or even set it to zero to allow shutting down the SMF service for however long it would take. This can make sense if for example your VM has a large virtual HDD or RAM, and simply takes long to spool to disk when doing "savestate".

Another is the timeout "vm/stop_timeout" which, if not zero, causes 'stop_vm()' to abort waiting for VM exit/savestate after a while and return an error state. This may be a smaller timeout than that configured for SMF (especially if that is not set =0), an by default this is 3600 seconds to allow for any reasonable long shutdown.

Last but not least, during an OS shutdown, the SMF framework itself can enforce some timeout after which all still-running processes are kill()ed. This timeout may be hardcoded in bootadm or in svc.startd programs. In effect this would ignore any other timeouts (if they are larger than, say, 5 minutes) and would lead to broken saved states if they take too long to write.

This can also lead to some problems i.e. when you have a networked setup with a storage server keeping image files for the VMs which are executed on another machine. For example, if the whole server rack is due to shut down because of an UPS on-battery event, your storage host might quickly shut down and leave all VMs hanging until killed (by timeouts or lost power) because they can not continue saving their states. I guess you would need some clustering framework to do shutdowns and starups in an orderly fashion safely in such scenario, or do a workaround like "storage host sleeps 2 minutes on shutdown request and shuts down; whoever of VMs was lucky to get saved - got saved, others reboot".

But this is likely not your case, since you are also interested in desktop usage...
dsisevran wrote:And how to launch the display of a VM when we are logging on the graphical interface of the host?
This was not a primary target for the service, because it is mostly server-oriented.
However, the method script has an option to attach the VM to X11 GUI interface - for debugging/maintenance purposes.

Technically it saves the VM state to disk, pauses the VM, and resumes it with another VirtualBox process. When you are done with the GUI, you have to save-state again, and the SMF service will resume the VM with a VBoxHeadless mode.

Note that if there are any problems with X11 startup (missing libraries in path, no "xhost" right to use the $DISPLAY, etc.) the VM-resume would fail and the VM itself is likely to crash (like in "reset").

I think you can make something like a desktop link or a command in your ~/.dtprofile to call the method script and attach the VM to your current X11 session.
I think it is up to you to carefully save-state the VM when you are logging out; otherwise, probably, the X11 GUI would get killed by the stopping X11 desktop session and the VM would die. I've never tried that myself so I am not sure.

Reports on any success or failures (and perhaps patches to the SourceForge) are welcome, though ;)

Thanks for using vboxsvc,
//Jim Klimov

UPDATED 2012-04-12: I've recently noticed that there is another feature to keep in mind when hosting VMs in local zones: the "zoneadm -z ZONENAME halt" command terminates the zone's processes quickly, without going through proper SMF shutdown procedures. For VM processes this quick termination would mean an abruptly aborted VM. Likewise for "zoneadm -z ZONENAME reboot".
The proper way to stop a local zone gracefully is "zlogin ZONENAME init 0", or "zlogin ZONENAME init 6". Docs for Solaris 11 mention that they added "zoneadm -z ZONENAME shutdown (-r)" for graceful shutdowns and reboots of local zones, but this is not (yet) available in illumos/OpenIndiana.
Default method script for stopping all zones (as "svcadm disable zones:default" in the global zone) tries the graceful shutdown for 3/4 of the configured stop timeout time, and then tries the ungraceful shutdown for the remaining 1/4 of the time. Finally, if this stop method does not finish stopping the zones, SMF in the would apparently kill the remaining zoneadmd processes and their children as it would on all timeout failures.
So, when hosting VMs in local zones, remember to use "zlogin...init..." to stop or reboot a zone, and expand the "zones:default" SMF timeouts to exceed the required and configured shutdown times of your VM service instances. Also remember that if many VMs need to save-to-disk, this would likely take longer in wall-clock time than your possible tests of "how long does it take this VM to suspend?" executed for single VMs at a time.
Last edited by jimklimov on 13. Apr 2012, 01:11, edited 1 time in total.
dsisevran
Posts: 2
Joined: 4. Nov 2011, 12:20
Primary OS: Solaris
VBox Version: PUEL
Guest OSses: Windows 2003, FreeBSD 8

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by dsisevran »

@jimklimov:
Thanks for your quick answer.

It work like a charme for me. The VM boot with the host but have a little trouble when shutting down the host because I have configure the SMF property "vm/stop_method" with value of "acpipowerbutton".
I didn't change the SMF property "stop/timeout_seconds" but my solaris seems to shut down faster than 120s. Nothing was logged when I reboot the host.

Will try later with the savestate option because the VM is a DC... :lol:
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by Technologov »

Technologov: Since this topic is popular across Solaris fans, and seems to be important service for many users, I decided to make it sticky.
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's - NOTI

Post by jimklimov »

Hello, happy users of the package! And thanks to the moderators for making this a sticky thread ;)

A misbehavior was found regarding auto-shutdown (or suspension) of VMs when the vbox service stops on some systems. It is actually rather an SMF problem (SMF implementations seem to vary in different versions of Solaris and OpenSolaris and its descendants - happens to bleeding-edge systems), but nonetheless, here goes:
If the administrator created per-instance settings for "stop" (and probably "start" - not tested yet) framework, such as defining the "stop/timeout_seconds" value custom for a service instance, some SMF implementations do not pick up "defaults" provided by the service-level definitions of the "stop"/"start" frameworks. In particular, this means that the stop/start methods are no longer defined for this SMF instance, and the method script is not even called.
This can be seen in the SMF logs with lines like these:

Code: Select all

# tail /var/svc/log/*VM_NAME*
...
[ Jan 17 16:30:29 Stopping because service disabled. ]
[ Jan 17 16:30:29 Method property 'stop/exec is not present. ]

Stopping the service instance (i.e. "svcadm disable -st VM_NAME") completes instantaneously, but the VM processes in fact continue to work uninterrupted - because the stop logic was never called.

If your system shows such behavior, a somewhat bulky workaround is to redefine all "stop/*" ("start/*") settings at the instance level as soon as you define any custom value. For example:

1) Test if your VM's service instance (called svc:/site/xvm/vbox:VM_NAME) has custom values for SMF "stop" framework:

Code: Select all

# svccfg -s VM_NAME listprop | grep stop/
stop/timeout_seconds               count    300
2) Test if stops work or fail:

Code: Select all

# time svcadm disable -st VM_NAME
real    0m0.010s
user    0m0.001s
sys     0m0.003s
### That was way too quickly for a proper stop!

# tail /var/svc/log/*VM_NAME*
...
[ Jan 17 16:30:29 Stopping because service disabled. ]
[ Jan 17 16:30:29 Method property 'stop/exec is not present. ]

3) List the SMF-service-level framework parameters:

Code: Select all

# svccfg -s svc:/site/xvm/vbox listprop | grep stop/
stop/type                                  astring  method
stop/exec                                  astring  "/lib/svc/method/vbox.sh stop"
stop/timeout_seconds                       count    220
4) Redefine all missing settings for your VM (note the quoting for parameters with spaces):

Code: Select all

# svccfg -s VM_NAME setprop stop/type = astring: method
# svccfg -s VM_NAME setprop stop/exec = astring: '"/lib/svc/method/vbox.sh stop"'
# svcadm refresh VM_NAME
# svccfg -s VM_NAME listprop | grep stop/
stop/timeout_seconds               count    300
stop/type                          astring  method
stop/exec                          astring  "/lib/svc/method/vbox.sh stop"
5) Reenable the service instance (NOTE: this might suspend and restart your VM):

Code: Select all

# svcadm enable VM_NAME
# tail -f /var/svc/log/*VM_NAME*
...
Hope this helps,
//Jim Klimov
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

New release 0.15 has been published today at https://sourceforge.net/projects/vboxsv ... -20120130/

Main new feature of this release is the ability of vbox.sh to create snapshots of ZFS datasets backing the VM and/or its data. Datasets containing the VM config files, VirtualBox snapshot files, and HDD image files, can be found automatically. Other datasets (such as raw ZVOLs) can be listed in a special SMF attribute. If enabled, this feature creates snapshots after stopping the VM and before starting it. Can also be called as a command-line action.

Other new features include the "poweroff" command-line action ("pressing" acpipowerbutton first, then calling poweroff if ACPI fails), and a timeout override for poweroff and reboot command-line actions.

Have fun!
//Jim Klimov
martyscholes
Posts: 202
Joined: 11. Sep 2011, 00:24
Primary OS: Solaris
VBox Version: PUEL
Guest OSses: Win 7, Ubuntu, Win XP, Vista, Win 8, Mint, Pear, Several Linux Virtual Appliances

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by martyscholes »

Jim,

Thanks for this amazing tool. I am using it to start VB instances in zones. Soon I will install the latest. I did notice a couple of things with the previous version. It's entirely possible I did something wrong or missed something in this thread, but wanted to run it by you.
* The service depends on some gnu stuff, specifically, gdate
* The service depends on the VB user being able to restart the service

Having this written down might help the next guy. Again, apologies if this has already been covered.

While not directly related to this service, it took me quite a while to navigate networking in a zone to get a nic (for use by the VB image) plumbed by the zone but not in use by the zone. I needed this because otherwise an ssh call to the VB image was intercepted by the zone, same for http, etc.

Thanks again for the service!
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Could you please elaborate on the second problem and setup? ;)
Requirement of gdate stems from a seconds-precision of timestamps, and it is noted in the readme and comments. I'm open to suggestions of more portable solutions... might port the perl one-liner from some of my other scripts, but it is just another dependency - on perl ;)

Regarding the VB non-root user, you can set up RBAC and SMF to allow that account to manage the service. Link to the docs was in the readme and in script comments.
Alternatively, the non-root user can trigger the fall of service into maintenance mode (if the internally-called svcadm restart command fails).

Can you please explain where these two methods don't suffice for you or could be made better? ;)

Regarding a zone with VNICs, I have a setup like that. Lots of VNICs are defined in global zone, attached to real links or to etherstubs for some internal virtual switches. The local zone is ip-type=exclusive (it acts as the router for VMs running in it and attachet only to etherstubs). VNICs are not plumbed in the global zone, and they are connected to the zone as lots of commands in the zone's XML manifest like this:

Code: Select all

  <network address="" physical="vnic127198"/>
  <device match="/dev/net/vnic127198"/>
Interfaces made for the zone itself are referenced as files like /etc/hostname.vnic163001 in the zone, the file contains IP address and other settings for this VNIC, i.e.:

Code: Select all

# cat /etc/hostname.vnic163001
192.168.163.1/24 mtu 1500
VNICs for VMs are also referenced by such files, but they are empty. As a result the VNICs get plumbed (but not UPed) by the local zone, and then bridged-on by the VMs.
This all works well ;)

Hope this helps you and others ;)
//Jim
martyscholes
Posts: 202
Joined: 11. Sep 2011, 00:24
Primary OS: Solaris
VBox Version: PUEL
Guest OSses: Win 7, Ubuntu, Win XP, Vista, Win 8, Mint, Pear, Several Linux Virtual Appliances

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by martyscholes »

it is noted in the readme and comments
Link to the docs was in the readme and in script comments
Ok, I probably should have read your documentation. My bad.
Can you please explain where these two methods don't suffice for you or could be made better?
I didn't mean to whine, but more to warn the next guy and say "look here."
Could you please elaborate on the second problem and setup?
I am still learning the whole ipadm vs. /etc/hostname.blah. If the zone has automatic networking and the second nic is provisioned, then the zone will plumb the second nic.

It all worked out. Thanks again!
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

I've made a new version which can fall back to PERL timestamping if GDATE is not available. I did not make a release with it yet - needs some testing (worked for me, and gdate is a bit faster). Possibly UTC-nonUTC timestamps may be mixed up. This may matter when evaluating lock-file ages; matters less for relative values (like X seconds ago) unless you're lucky to have the summer-winter nonUTC time jump. I hope that all timestamps involved are produced in UTC.
You can try to download it from sourceforge repository: http://vboxsvc.svn.sourceforge.net/view ... revision=6
Tell me if it works for you ;)
//Jim
Napalm_God
Posts: 9
Joined: 15. Dec 2008, 01:22

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by Napalm_God »

Hi jim,

I have a problem in creating instances for some of my VMs.
Several of them have spaces in their name, so when I define them in the xml file, e.g.

Code: Select all

   <instance name='scrap machine' enabled='true'>
what happens is that issuing

Code: Select all

#svccfg validate test_manifest.xml
all goes well, but when I try to import the manifest

Code: Select all

#svccfg import test_manifest.xml
I get the following error:

Code: Select all

svccfg: Service "site/xvm/vbox" has instance with invalid name "scrap machine".
svccfg: Import of test_manifest.xml failed.  Progress:
svccfg:   Service "site/xvm/vbox": not reached.
svccfg:     Instance "scrap machine": not reached.
svccfg: Import failed.
I get no errors and can successfully start any VM that has no spaces in its name.
So, must I resolve to rename all of my VMs, or is there a way to import them as they are (with spaces in their names)?

Excuse me in advance for my noobness...
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Napalm_God wrote:Hi jim,
I have a problem in creating instances for some of my VMs.
...
I get no errors and can successfully start any VM that has no spaces in its name.
So, must I resolve to rename all of my VMs, or is there a way to import them as they are (with spaces in their names)?
Hello, I haven't really looked into VMs with spaces in names, as well as this is (I think) not a valid character for SMF service names.

In general, the UNIX way is avoiding popular/possible separator/comment/wildcard characters (spaces, tabs, end-of-lines, slashes, colons, hash-signs, asterisks, quotes and lots more) in FS object names Even though technically only two chars are forbidden - the directory separator (fwd slash) and end-of-string (null-character).

I think it might be possible to add another non-required attribute to the SMF manifest - VirtualBox VM name, in case it differs from SMF instance name. The feature might be useful regardless of the spaces-in-names problem, but some care and testing should be in place to make sure this would actually work (i.e. iterations like "for VAR in $LISTVAR" are likely to break if components of $LISTVAR also have spaces inside). Beside that, FS paths with spaces may also be processed incorrectly by my script as well as other programs it calls, much for the same reasons.

Overall, I think that a faster and safer solution would be indeed to rename your VMs (and FS paths) using dashes or underscores instead of spaces, so that each name becomes a single token.

HTH,
//Jim Klimov
Napalm_God
Posts: 9
Joined: 15. Dec 2008, 01:22

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by Napalm_God »

Yeah, I guess I'll just rename my machines.
I actually created most of them on Windows, that's why the names contain spaces.

Oh, well... time to fix that up! ;)
sjorge
Posts: 4
Joined: 12. Apr 2012, 22:47

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by sjorge »

I'm having some problems with USB devices not being available (in use) at system boot so starting of some machines fails.
Is there a way to put a delay in starting of a vm? Say wait 2 minutes before starting it?
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

This was not catered for "out of the box", but I can think of a few solutions:
1) Change the vbox.sh script (or create a wrapper script to call vbox.sh passing all arguments) to "sleep 120" in the start method, and update the SMF instance/service start timeout to be larger (at least 3 minutes for example)
2) Same as 1, but instead of hard-coding the sleep value, expand the script to make this a per-VM configurable variable (with a service-wide default being zero, but configurable) and submit the patch ;)
3) If you only need this during a startup, create a dummy service with a service type "transient", whose start method would be "sleep 120". After the start method succeeds, the dummy service will be onlined and its dependencies can start. Since it would be a transient service, SMF doesn't track whether there are any processes or failures after the start method exits with zero result code (success). I think for safety you'd need a slightly larger start timeout for the dummy service, say at least 130 seconds, so that the startup is not aborted. Then configure the vbox service instances to depend on that dummy service (I think it might also be easier defined in the dummy service as "services known to depend on me"), without changing the vbox SMF timeouts and/or inserting sleeps into vbox startup.

I'd personally go for method (3) as the most transparent one and requiring no custom changes to vbox.sh script, so that it can be safely upgraded in case of new vboxsvc releases. Also in this case the vbox SMF restarts during your normal work would not incur extra delays.

If you're new to Solaris and SMF, this might be a useful learning experience. If it is still to hard, I or other community members might come up later with a sample SMF manifest for the sleeping dummy service...

HTH, and thanks for using vboxsvc,
//Jim
Post Reply