[Free as in beer] SMF service for VirtualBox VM's

Discussions related to using VirtualBox on Solaris hosts.
sjorge
Posts: 4
Joined: 12. Apr 2012, 22:47

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by sjorge »

Hi Jim,

Thanks for the tips, option 3 does look the most interesting!
I give it a go but as you assumed correctly my knowledge of Solaris is fairly limited for now.

Here is what I have so far... but I think the dependent is the part that is failing.
Some help would be appropriated but in the main time I'll keep hammering at it.

Now works... But still usb errors. Manual starting vm directly afterboot works as expected :s

Code: Select all

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<service_bundle type='manifest' name='vbox-needs-usb'>

<service
    name='site/xvm/vbox-needs-usb'
    type='service'
    version='0.1'>
    <create_default_instance enabled='false' />
    <single_instance />


   <dependent name="vbox" grouping="require_all" restart_on="none">
      <service_fmri value="svc:/site/xvm/vbox"/>
   </dependent>

   <exec_method
      type='method'
      name='start'
      exec='sleep 240'
      timeout_seconds='250'
   />

   <exec_method
      type='method'
      name='stop'
      exec=':true'
      timeout_seconds='60'
   />

  <property_group name='startd' type='framework'>
    <propval name='duration' type='astring' value='transient' />
  </property_group>

</service>
</service_bundle>
This also inspired me to create another script; My gateway is run from vbox... so solaris itself has a few services that fail to start most notably system-repositories. This one fixes those issues.

Code: Select all

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<service_bundle type='manifest' name='vbox-provice-gateway'>

<service
    name='site/xvm/vbox-provice-gateway'
    type='service'
    version='0.1'>
    <create_default_instance enabled='true' />
    <single_instance />

   <!-- wait for vbox to be started -->
   <dependency name='vbox' type='service' grouping='require_all' restart_on='none'>
      <service_fmri value="svc:/site/xvm/vbox"/>
   </dependency>

   <!-- if a vm provides the gateway, some services fail -->
   <dependent name="system-repository" grouping="require_all" restart_on="none">
      <service_fmri value="svc:/application/pkg/system-repository"/>
   </dependent>

   <!-- wait 2 minutes to give the vm time to start -->
   <exec_method
      type='method'
      name='start'
      exec='sleep 240'
      timeout_seconds='250'
   />

   <exec_method
      type='method'
      name='stop'
      exec=':true'
      timeout_seconds='60'
   />

  <property_group name='startd' type='framework'>
    <propval name='duration' type='astring' value='transient' />
  </property_group>

</service>
</service_bundle>
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Hello sjorge, it is nice to see the idea evolve into something working so quickly ;)

I think that the problem with your first manifest may be that it might start too early in the system boot, so its 2 minutes elapse while things are still booting and initializing. Try to add a dependency on some OS milestone, like "network" or "all", so that the sleep begins after the boot-milestone has been reached, and the vboxes start even later on.
Also I see that in the manifest the default instance is disabled, I hope you did enable it later on? ;)

The second manifest is a wonderful example IMHO of using SMF to straighten things out without direct modifications of other (third-party) services. Kudos for that idea! ;)

Still, you might want finer-grained control, i.e. instead of depending on "vbox" in general, set a dependency on the gateway VM's SMF instance in particular, for the 'vbox-provice-gateway' service.

Likewise, if the gateway (and/or other VMs) does not need the USB workaround, you might specify only those VMs (SMF instances) that need USB as dependents of the sleeper 'vbox-needs-usb'.

Good luck, and keep us posted,
//Jim

PS: And 2 minutes are 120 seconds, not 240 as in your examples ;)
But the required real-life delays/lags are what matters and what you configure here.
sjorge
Posts: 4
Joined: 12. Apr 2012, 22:47

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by sjorge »

edit: Back to KISS, one script do all is complex and not really working

vbox-helper.sh

Code: Select all

#!/usr/bin/bash
## configuratiom
sleep=$(svcprop -p "helper/sleep" "$SMF_FMRI" 2>/dev/null)

# main
sleep ${sleep}
vbox-provices-gateway.xml

Code: Select all

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type="manifest" name="vbox-provides-gateway">
	<service name="site/xvm/vbox-provides-gateway" type="service" version="0.3">
		<create_default_instance enabled="true" />
		<single_instance />

		<!-- make us run BEFORE vbox:* -->
		<!-- change instance to gateway -->
		<dependency name="gateway-vm" type='service' grouping="require_all" restart_on="none">
			<service_fmri value="svc:/site/xvm/vbox:inertia"/>
		</dependency>

		<!-- do some reordering since we provide network connectivity -->
		<dependent name="repository" grouping="require_all" restart_on="none">
			<service_fmri value="svc:/application/pkg/system-repository:default"/>
		</dependent>
		<dependent name="zones" grouping="require_all" restart_on="none">
			<service_fmri value="svc:/system/zones:default"/>
		</dependent>

		<!-- do the heavy lifting... well waiting -->
		<exec_method
			type="method"
			name="start"
			exec="sleep 1"
			timeout_seconds="60"
		/>

		<exec_method
			type="method"
			name="stop"
			exec=":true"
			timeout_seconds="60"
		/>

		<!-- mark service as transient -->
		<property_group name="startd" type="framework">
			<propval name="duration" type="astring" value="transient" />
		</property_group>

		<!-- more stuff about this service -->
		<stability value="Unstable" />
		<template>
			<common_name>
				<loctext xml:lang="C">VBox Gateway Provider</loctext>
			</common_name>
		</template>
	</service>
</service_bundle>
USB issue was not related to the availability at early boot.
Booting with it plugged in... made the 3G Modem appear as a storage device and crash.
I fixed it using:

Code: Select all

update_drv -a -i 'usb12d1,1001' usbsacm
This was for a HUAWEI device it it will mostlikely not work for others.

I'm still keeping the vbox-provides-gateway service, it prevents my zone-proxyd and system-repositories from going into maintenance mode.

Hopefully someone will find these usaful also, I'm already thinking of some alternative uses, small edits could make you set a fixed boot order with timeout values. e.g.
vbox:fw1 -> vbox-provices-gateway -> zones/repostories
vbox-needs-fw1 -> vbox:srvad -> vbox-needs-srvad -> vbox:cliw7

can be done with editing the script and setting the correct dependancies, you can also specify a wait time that is different. Sure it is a bit hacky, but as mentioned above.
No edits where done to ANY systems service or vbox-svc.

I'd love to parse the vbox-needs-xyz part and auto set the dependancies at runtime but thats not possible :( {or I'm not skilful enoufh, this is my first SMF manifest after all}
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Well, I think something can be done for more reliable SMF dependencies by evolving my script: it already allows configuration of "monitoring hooks" - host-side user-made scripts which should check that the VM presents its services wanted by the user (i.e. SSH or Webserver or PING responds).
These hooks are currently implemented to check the VM at runtime (i.e. if virtualized services fail for some times in a row/timeframe, host can reboot the VM and/or put the SMF instance into maintenance).

I'm now thinking of a new feature to use these hooks (if configured) to also wait for VM startup to produce a positive result for the hook check, and only exit the start method at this time. I had that in mind, but the implementation was indefinitely delayed ;(

This might be coupled with a configurable delay after which to exit the start method (perhaps inferred from SMF start timeout minus the time it took to process the call to VirtualBox), and configurable policy:
0) exit OK as soon as the VM has started and the KICKER loop began (as is now)
1) exit OK after the timeout if the hook is not configured for this VM instance
2) exit OK after the timeout regardless of the (latest) hook test results if hook is configured, and just let the VM continue its booting as well as allow startup of VMs that depend on it.
3) exit OK after the timeout or FAIL (maintenance) if the latest hook test failed - but this can lead to somewhat false negatives if the VM just takes longer to boot for any reason (resource contention, etc.), so proper timeout configuration would be needed from admins
(4) Actually, if the configured hook tests succeeded before the timeout, the start method should exit OK then... unless the user really also wants the verbatim startup delay to be in place.

If such features make it into vbox.sh, then we won't need the dependency services which just sleep to add delay (and don't even check that the booted VM actually works). Your SMF instances like "vbox:cliw7" would just depend on your "vbox:srvad" instance. Its method script should then take several minutes to complete, but when it does (successfully with policies 3 or 4) - the VM is known to be fully functional according to the hooked test scripts which you defined.

One possible problem is that test scripts can run indefinitely long, ultimately exceeding the startup timeout, but I can think of several workarounds to that.

Do you think that would solve the problem for you (of actually having services and VMs which depend on each other)?
Would you mind if I incorporate your post-boot-delay manifest into a future release of vboxsvc as an optional feature? ;)
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Well, thanks to this weekend's discussion, a new release vboxsvc-0.16 has been written, somewhat tested on my systems and released at https://sourceforge.net/projects/vboxsv ... -20120415/

Probably the main new feature of this release is the ability of vbox.sh SMF start method to delay its exit by either a timeout or until the KICKER monitoring script succeeds (if before the timeout). This allows to more correctly start VMs in the order that one's services are needed by others by configuring SMF dependencies, startup timeouts and monitoring scripts (note: proper setup of your SMF dependencies was not included in the generic package ;) ).
To enable such exit delays (disabled by default) there is a new service attribute 'vm/start_wait_policy'. This 'astring' can accept a few key values:
* nodelay - Do not incur extra delay after VM startup. This is the default in case of invalid settings.
* timeout - Only wait for the (start/timeout_seconds-5) to elapse and then complete the start method with OK.
* hook_strict - If vm/kicker_vmsvccheck_method and kicker_vmsvccheck_method_params are set correctly, run tests on the VM and wait for positive results. If the hook results are not OK by the time that timeout expires, shut down the VM and fail SMF instance into maintenance.
If the tests succeed before timeout, exit OK.
If kicker_vmsvccheck parameters are not set, act as the timeout (and ultimately exit OK).
* hook_lax - Relaxed hook checks like above: if the tests do not succeed before the startup timeout, note this in the log but exit OK, allowing the VM to continue booting.

Also included is an SMF manifest for a sleeping dummy service to delay the vbox services (all or per-instance) startup after the host OS boot.

Another recently requested feature is the lack of firm requirement for GNU gdate. Perl can now be used instead by the script, and it is more widely available as part of the basic OS requirements (though slower for this task of determining timestamps). The gdate binary will be used if discovered.

Thanks to everyone for using the package and keeping in touch, the feedback is much appreciated!
//Jim Klimov

PS: It would be nice if someone also posted the first comment or review on the SourceForge page. And the second would be nice. And some more... The project is some 16 months old with 800 downloads, and nobody liked or disliked it officially! ;)

PPS: The download stats are also an interesting benchmark to see where Solaris admins live and sit for day-to-day tasks... Roughly 35% each were using Solaris and Windows, another 10% each used Linux and Mac, and others were undefined. Note that this is a Solaris-only relevant project :)
sjorge
Posts: 4
Joined: 12. Apr 2012, 22:47

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by sjorge »

Hi Jim,

1) that would indeed fix my problem.
2) yes you can include the sleeper
3) will play with the new script once my data migration is done... seem it will take a while but should be able to bring the box up and down a few times after that.
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Today I released version 0.17 - it was brewing in the unpackaged development stage for a year too long. As usual, there are two deliverables:
* a simple archive - http://sourceforge.net/projects/vboxsvc ... z/download
* a SVR4 package - http://sourceforge.net/projects/vboxsvc ... z/download
The browsable source code should be available here: https://sourceforge.net/p/vboxsvc/code/43/ (after SF syncs its nodes) or http://svn.code.sf.net/p/vboxsvc/code/ (current revision)

New release brings a number of features, including the following Updates in release 0.17 (2013-05-10) noted in the README file:
  • * Added flags "vm/is_interactive" and "vm/is_dualboot" to assign different defaults to some other "auto-guessed" flags if they are not defined in this VM's SMF instance (and disable inheritance from service level).
    * The getState() routine (i.e. the status command-line option) now checks results of the monitoring hook if configured (outputs test results, but does not influence return code), and returns more "ps" listing info.
    * Added command-line action "vmsvccheck" to run the monitoring hook script if configured for the VM, and return its result code.
    * Added command-line action "start-force" to enforce (headless) startup of a non-running VM under SMF regardless of its many autostart=false SMF properties and possible is_interactive/is_dualboot settings
    * Added "start-gui" as (undeclared) alias to "startgui"; upon starting the GUI the script should also tell how to "return" the VM to SMF and its curent SMF state and relevant properties.
    * Added option "startgui -fg" to remain in foreground, and many modifiers for it, including:
    • "-stop_method" to override default or SMF-configured method of stopping the VM;
      "-t" to override timeouts;
      "--*" are passed to VirtualBox itself (--fullscreen, --seamless, etc.).
    The foreground variant of "startgui" also sets up a trap to catch the common termination signals and properly shutdown or savestate the VM when the calling terminal is closed (i.e. X11 closing); this should be good for safe use of dualboot/interactive VMs (startgui from dtprofile). The list of signals to trap can be configured in SMF service properties.
    NOTE: as of now, when an X11 session is logging out, the server doesn't seem to send any OS signals to its clients - it just closes connection and most of its clients just die ungracefully (VirtualBox GUI does). This is an upstream problem with X11 and Oracle VirtualBox. In order to gracefully savestate or poweroff an interactive VM running via "startgui -fg", YOU should close the controlling terminal which started the VM GUI program; perhaps tying this into the logout routine.
    I posted bugs upstream and now think about possible workarounds...
    * Added support for simple invokation of "startgui -fg" with symlinks to the main script, that can be used in GNOME panel launchers. This mode parses the script's used filename and determines the VM and optional stop method and stop timeout to be used for this launch. Example symlink:
    • vbox-startgui:ubuntu:poweroff-graceful:30 -> /lib/svc/method/vbox.sh
    As a result, an xterm is launched with vbox.sh invokation. By closing that terminal window you can gracefully stop or save the VM as requested, i.e. do this just before you log out of X11.
    * Added SMF properties to disable auto-start of halted (poweroff'ed) and saved VMs while the service is considered enabled. This should aid in manual startup of some VMs (i.e. interactive GUI ones on desktops) and still provide proper automated shutdown via SMF along with the host OS.
    * Added a vm/stop_method value "poweroff-graceful" -- this first tries "acpipowerbutton" with the defined vm/stop_timeout (keep note to define it to be under SMF stop/timeout_seconds); if that doesn't succeed - retries with brute "poweroff".
    * Added a wrapper for 'socat' coupled with detection of VM's UART serial ports in VM configs, in order to easily connect to serial consoles.
    * The zfssnap method now accepts optional tag text for the snapshot, to be used instead of the default "manual snap" string.
    * Added the experimental zfs-zvolrights script which can help on VM/Storage hosts which use ZFS volumes to back VM images and run VMs with unprivileged user accounts. More detail in its own README file.
As usual, there is a lot more planned to be done, subject to available time and inspiration. Have fun, and I hope this helps you use your Solaris/illumos-based virtualization hosts more efficiently,
//Jim Klimov
Post Reply