[Free as in beer] SMF service for VirtualBox VM's

Discussions related to using VirtualBox on Solaris hosts.
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

[Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Post updated by Jim Klimov, 2010-08-27:
---

Hello, all.

Like many others, I wanted to run some VMs all the time regardless of host reboots.
And on Solaris/OpenSolaris hosts this is facilitated by SMF services.

Almost a year ago Alexandre Dumont published his SMF method script and manifest, and they allowed to do just that: register each VM as an instance of the SMF service and enable or disable it. It also allowed management of VMs owned by a non-root user (via service credentials) and, of course, SMF allows to set up dependencies between VMs, resources and other services.
http://adumont.serveblog.net/2009/09/01 ... box-smf-2/

Over the past months I've used and extended this service:
* to enable management of paused VMs,
* savestate and restore the VMs (i.e. on service disable/enable),
* run certain VMs with a tweaked NICE priority,
* reboot VMs gracefully via acpibutton-poweroff-poweron or ingracefully reset actions,
* monitor the VM state according to VirtualBox (i.e. aborted, saved, etc.) and appropriately reflect this in SMF service state (cause maintenance or offline state), or start up the VM,
** can count how many aborted states there were over the past X seconds and cause maintenance for a frequently failing VM (i.e. when the NFS sever with its virtual disk is down, etc.),
* a number of command-line modes for the manifest script, including the ability to save VM state, restart it with a GUI mode for manual maintenance, and when you're done - save state again and the VM will be managed by its SMF service again,
* UNTESTED hooks to call external scripts which can poll the VM's services to see if it is alive inside.

Updates in release 0.11 as a difference from previously published 0.09:
* Add processing for VBox empty state as a temporary error (retry next cycle),
* Added a flag to cause all 'offline' attempts to actually cause 'maintenance',
* Use `id` invokations good for both OpenSolaris and Solaris 10,
* add more definite dependencies for SMF service startup (milestone multi-user, filesystems/local; nfs/client and autofs are optional = required if enabled) as a result of debugging our occasional failures with VMs running off NFS storage,
** for users implementing iSCSI this should probably provide a template to require iSCSI clients to start before VMs.

All thinkable variables have been parametrized with SMF service properties (group "vm/" or system props in groups "start/", "stop/", etc.), and properties not defined at the instance level (i.e. "svc:/site/xvm/vbox:VM_NAME") will be queried from the common service level (i.e. "svc:/site/xvm/vbox").

Feel free to test this in your environment, and please report back whether this worked well or not, or what can be fixed/improved. Patches are welcome ;)

Hopefully this would not ruin your system, but beware that some PID- and LOCK-files are used: created and removed. Some of this activity can be enabled or disabled per-instance, any filenames used can be hardwired into SMF properties (must be unique!). See comments in XML manifest for more details.

Hope this helps,
//Jim Klimov
Attachments
vboxsvc-20100824.tgz
Just the sources, updated. They go into:
/lib/svc/method/vbox.sh
/var/svc/manifest/site/vbox-svc.xml
(15.15 KiB) Downloaded 349 times
COSvboxsvc-0.11.pkg.gz
Packaged variant for simplified installation, updated
(15.84 KiB) Downloaded 477 times
Last edited by jimklimov on 27. Aug 2010, 19:24, edited 1 time in total.
bqbauer
Posts: 159
Joined: 14. Dec 2008, 22:04

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by bqbauer »

Although I'm more interested in using VB as a desktop oriented service, I've gotta try this. You never know what I might learn or how I might find this useful. Thanks for the info!
tomwaters
Posts: 20
Joined: 26. May 2010, 00:31
Primary OS: OpenSolaris 10
VBox Version: PUEL
Guest OSses: Windows 7

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by tomwaters »

Sweet. Thanks for sharing.

I have been using the original (slightly modified) script to auto-start my VM's for some time now and it works a treat!

Combine this with some icons to restart/stop/remote desktop and I have been a very happy user.

The cpu utilisation is much lower as a headless VM so I urge everyone to give this a try...it's very easy to setup and manage.

Thanks again for sharing.
pressingonalways
Posts: 2
Joined: 27. Aug 2010, 21:16
Primary OS: Solaris
VBox Version: PUEL
Guest OSses: Win2k3, Linux

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by pressingonalways »

Hi,

Can you post a howto on how to use this? Looks like it has many features, but I am not too familiar with SMF services.

So far I discovered you have to edit the /var/svc/manifest/site/vbox-svc.xml to match your username and group and then the machine name, then do

Code: Select all

pfexec svccfg import /var/svc/manifest/site/vbox-svc.xml     
after that though, then what?

svcadm enable vbox:VMName works, but I see there are hooks to capture from headless to GUI, how do you use all that?

Thanks,
Jonathan
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

I was trying to put together a howto for some time, but the scripts evolved too fast to document single-handedly, so I dropped those attempts. Can try to summarize them here as a starter, though.
But probably the best documentation (at least most up-to-date) is the code and the lots of embedded comments, hopefully readers would find them readable and clean :)

Here we go with a mostly SMF-oriented part of the howto, as a series of posts (got quite long).

1) "COSvboxsvc" Overview

This updated "COSvboxsvc" comes as an SVR4 package file, and as a couple of files - "method script + xml manifest".
Package file contains the same two important files plus some metadata to streamline script-version updates, and to keep track that the script is indeed installed in a certain zone. It also declares a dependency on SUNWvbox package.

Either way it can be used for running VMs in global or local zones, owned (and executed) by "root" or less privileged users.
Presumably, if you have set up the VM to do whatever you want from under the "/opt/VirtualBox/VirtualBox" GUI - setting up all needed RBAC and filesystem permissions, unique MAC addresses or console RDP ports, firewall rules or whatever else can limit your VM's usability, this script can manage such VM if executed with the same credentials via SMF.

I've run it only with Solaris 10 (updates 6 and 8 ) and OpenSolaris SXCE ("Nevada") snv_117 thru snv_129 (x86 and x86_64).
I haven't tried it (or VirtualBox) with OpenSolaris Indiana (i.e. osol_b134) or any other distribution, so I can't guarantee how it goes there. As far as I know, it is now "mauvais ton" to run programs (including shell) as a root, so all root-executed command lines are prefixed with "pfexec", while the shell is unprivileged. Sun docs formulated this as "become superuser or assume an equivalent role".

Now, while the VM processes can be executed as an unprivileged user, it takes a certain set of privileges to control an SMF service.
For this reason, the SMF method script for a VM running with credentials of a non-root user can not (by default) change state of its own SMF service instance (i.e. to initiate "offline" state for a temp-failed service). If this happens, script tries to do its best to cause SMF "maintenance" state by making a lockfile to prevent subsequent restarts.

Proper privileges can be delegated to a user account via RBAC profiles. See docs, i.e. http://hub.opensolaris.org/bin/view/Com ... up+smf/faq chapter 2.1.

2) Installation

a) Package format, global zone:

Code: Select all

# gzcat COSvboxsvc-0.11.pkg.gz > /tmp/x
# pkgadd -d /tmp/x -G
You probably want the -G flag. It doesn't block you from manually installing the same package in a local zone, but it blocks propagation to local zones which are not expected to use and run VirtualBox. For us these zones are rare, zero or one per machine (there is no definite/hardcoded limit though). YMMV.

To update the package you can simply remove the old version and install anew, i.e.:

Code: Select all

# gzcat COSvboxsvc-0.11.pkg.gz > /tmp/x
# pkgrm COSvboxsvc
# pkgadd -d /tmp/x -G
A cleaner way is to use an admin file, i.e. one from LiveUpgrade:

Code: Select all

# gzcat COSvboxsvc-0.11.pkg.gz > /tmp/x
# pkgadd -d /tmp/x -G -a /etc/lu/zones_pkgadd_admin
Also note that this package "depends" on SUNWvbox, so that should be installed beforehand.

b) Package format, local zone: like the global zone, but without the "-G" flag, i.e.:

Code: Select all

# gzcat COSvboxsvc-0.11.pkg.gz > /tmp/x
# pkgadd -d /tmp/x
c) Files. Copy script to */lib/svc/method/vbox.sh* and the XML file - to anywhere you can edit it.
My package places it into the */var/svc/method/...* tree so it will be automatically imported after zone's reboot (updating the SMF repository as needed by XML file's version tag), but this is not a requirement if you plan to edit and import the file manually anyway.

to be continued...
Last edited by jimklimov on 30. Aug 2010, 18:51, edited 1 time in total.
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

3) Working with SMF service

You'd best read up the original documentation. Some random pointers:
* Sun BigAdmin: http://www.sun.com/bigadmin/jsp/utils/P ... _intro.jsp
* Sun Wiki: http://wikis.sun.com/display/BigAdmin/SMF+Short+Cuts
* Joyent Wiki: http://wiki.joyent.com/solaris:smf-manifest-recipes

As an overview of what's relevant to COSvboxsvc: there is a concept of a "service" and its "instances".

a) Instances define some unique set of parameters ("properties") to run a specific VirtualBox VM. At the very minimum, they point to VM name (as spelled in *~/.VirtualBox/VirtualBox.xml* registry file) - as the instance's name, and often point to the user account whose credentials will be used to execute the VM. The same user should usually have access rights (read and/or write) to the VM files and directories ;)

b) Service definition groups together the OS concept of several virtual box VMs. It points, for example (via stop/start methods) to the *vbox.sh* script, or defines start/stop timeouts, or lists default dependencies. In our case the "service" is mostly a container of default configuration settings - but this is not the only use of "services" in SMF (see docs for discussion of different implementation of "smtp" service via "sendmail" or "postfix" instances, for example of other uses).

We can also define all of the parameters at the service level and they will be inherited by instances if not defined at instance level - but, again, this is not a generic SMF feature (vboxsvc was specifically coded to do this inheritance).

As may be outlined below, and is best described in code, many features (particularly those which deal with lock files) can be enabled or configured (i.e. to use definite filenames), for example to avoid some sort of DoSing or other security flaws with the default configuration (a well-known static config may fall prey to script-kiddies). Customized configuration provides a weak defense against that, but one thing for certain: these and similar properties, if configured, should be defined at the instance level. The same lock-file name should not be used for each VM, as inherited from service-level properties.
NOTE: most properties also have defaults hard-coded or evaluated during vbox.sh execution. For example, if lock-file names are not defined in SMF properties, they will be generated based on VM instance name.

c) You can start services. This checks all dependencies, and if satisfied, executed the start method as defined in SMF. In our case, calls "vbox.sh start"
# svcadm enable VM_NAME

d) You can stop services. In our case, this calls "vbox.sh stop" which, by default, saves VM state to disk for a quick restart (not VM reboot - if all goes ok) afterwards.
# svcadm disable -st VM_NAME

The "-s" flag causes "svcadm" to wait for the stop method to complete before returning to shell.
The "-t" flag causes this disablement to be temporary. If the service was previously "enabled" by svcadm, it will restart when the OS (or zone) reboots.

e) You can restart services. This is not often useful in our case, except to check (via log files) that the framework is set up correctly, or to apply new SMF properties. I use this a lot during development, so I also present how to monitor the log file :)
# tail -f /var/svc/log/site-xvm-vbox\:VM_NAME.log &
# svcadm restart VM_NAME

f) You can view and modify the service (or instance) properties. There are several ways: command-line snippets with svccfg or its shell-like interface with the same keywords, and XML-file edition and import. The XML file which describes all attributes of the service and its instances (like the one provided with this package, but without good comments) can be exported from the running system, edited and imported back into the system.

svccfg interface is mostly oriented for shell-like execution (run it and type "help"), but many of its commands can also be used as command-line constructs for single-command actions.

Two examples below show how to set individual properties for service or instance levels with command-line:

Code: Select all

# svccfg -s vbox setprop vm/offline_is_maint = true
# svccfg -s VM_NAME setprop vm/restart_saved_vm = true
For individual VMs (instances) you can override existing settings or add some dependencies, etc. This often requires you to do some research and define correct additional "property groups" and then define whichever properties you need.
For example, to set SMF timeouts, you need to add the "start" property group with type "framework", and then define and set some properties in this group:

Code: Select all

# svccfg -s VM_NAME addpg start framework
# svccfg -s VM_NAME setprop start/timeout_seconds = integer: 120
# svccfg -s VM_NAME setprop stop/timeout_seconds = integer: 240
For XML-file manipulation, you do something simple like this:

Code: Select all

# svccfg export vbox > /tmp/vbox-svc.xml
# vi /tmp/vbox-svc.xml
...
# svccfg validate /tmp/vbox-svc.xml
# svccfg import /tmp/vbox-svc.xml
You're encouraged to use the XML file approach to define additional new VM instances. The XML file provided with the package contains all defined properties (as processed by current script version) with explanations and comments, and it also contains a disabled instance called "VM_NAME" intended for copy-pasting for your VMs. Most of the definable properties are commented away in this block, so that service-level or hardcoded/evaluated defaults would take place, and instance definitions are quite short.

My typical VM instance definition looks like this:

Code: Select all

   <instance name='my-desktop' enabled='true'>
      <method_context working_directory='/var/tmp'>
        <method_credential group='staff' user='jim'/>
      </method_context>
      <property_group name='vm' type='application'>
        <propval name='kicker_blockfile_enabled' type='boolean' value='true'/>
        <propval name='start_aborted_vm' type='boolean' value='true'/>
        <propval name='start_paused_vm' type='boolean' value='true'/>
        <propval name='stop_method' type='astring' value='savestate'/>
      </property_group>
   </instance>
g) Refresh services. After you change the service or instance properties (via svccfg above), you should "refresh" each involved service instance. AFAIK this copies the properties from a staging SMF repository to an active one, and alerts service instances via a defined "refresh" method, if any.

In our case it is not defined, so defaults to ":true" and does nothing. Now, most of the properties involve run-time configurations such as flag file names or max abortion counts or the default stop method. If these properties are used in the KICKER loop (see code), they are reloaded from SMF on each cycle, so no actual refresh is needed, and ":true" is okay.

Nearly the only practical property which would require a service restart is the execution user account (method_credential); you use "svcadm restart" in this case...

To refresh all instances after property changes, do something like:

Code: Select all

# for S in `svcs -a | grep vbox | awk '{ print $3 }'`; do svcadm refresh "$S"; done
4) Working without SMF service

While *vbox.sh* implements some SMF service requirements (allows usage as an SMF start/stop method, uses SMF error codes and properties for config, etc.) it can also be used as a command-line script.

Usage for the current version can be viewed with the "-h" parameter:

Code: Select all

# /lib/svc/method/vbox.sh -h
To actually use the declared command-line methods on a specific VM, you must export the SMF_FMRI environment variable with the complete service instance name, i.e.

Code: Select all

# SMF_FMRI="svc:/site/xvm/vbox:VM_NAME" /lib/svc/method/vbox.sh startgui
Currently there are such methods as:
* getstate|state|status - summarize VM state as returned by VirtualBox query state, and SMF instance state.
* reboot - try to gracefully reboot the VM
* reset - try to quickly reset the VM
* startgui - save the VM state, and restart it with the GUI-mode VirtualBox (in VNC console to the headless host servers, in my case). If all goes OK, the VM resumes with the graphical VM console. You should test this before production, because this may lead to aborted VMs, i.e. if X11 is under-configured on a specific host.
When you're done with the graphical access to the VM, go to its window menu and select Machine / Close /Save state. When the VM is saved, the command-line-mode vbox.sh script exits, and the VM resumes as an SMF service instance automatically.
I confess this has failed on me more than once on some hosts (usually too minimized to run VNC and X11 in a stable manner), but also worked conveniently and quite well on many other hosts.

Hopefully this is all the *generic* info there is to say about COSvboxsvc and SMF services, and actualized details - such as property names, types and purpose - should be seeked in current code's/manifest's comments. Maybe (no promise) that will also be explicitly documented in a later post...

HTH,
//Jim Klimov
pressingonalways
Posts: 2
Joined: 27. Aug 2010, 21:16
Primary OS: Solaris
VBox Version: PUEL
Guest OSses: Win2k3, Linux

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by pressingonalways »

Thanks!! That is plenty of information for me to get started on! I appreciate especially the part on how to use

Code: Select all

 # SMF_FMRI="svc:/site/xvm/vbox:VM_NAME" /lib/svc/method/vbox.sh startgui
That was the part I was mainly clueless on...

Thanks!
Jonathan
jobo
Posts: 1
Joined: 28. Nov 2010, 13:54
Primary OS: OpenSolaris 11
VBox Version: PUEL
Guest OSses: Ubuntu, Debian, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jobo »

Hello,

i love very much the scripts you have posted here. Last year i did some research on that topic and was quite disappointed. Meanwhile i did not have the chance to play with opensolaris and vbox. Now I just got back to that topic. I was amazed, that you pushed this script so far.

Do you plan to develop, document and publish in a more systematic manner? I think of a hosted project like github, googlecode or something.

Thanks,
Jobo
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

I am more inclined toward SourceForge for open-sourced projects, but I did not yet pursue publishing it that far :)
Not that I have many ideas what else could be done beside docs...

One feature I thought of recently, but haven't yet implemented, is the possibility to pre-set TIMEZONE variables before executing VMs to, for example, "UTC" (or whatever was configured) - to have predictable local time on virtualized servers not dependent on executing user's timezone (which would likely become VM's hardware time).

But that's about it - of all the brainstorming and daily use for a few months now...

Thanks for liking it :)
//Jim
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

As requested, the "vboxsvc" project has been initially published on SourceForge:
https://sourceforge.net/projects/vboxsvc/

Feel free to add bug reports and patches to the tracker.

I'll be largely offline for a few months coming, though. So if anyone wants to be added as a co-developer to evolve the package (i.e. merge other siblings of Alexandre Dumont's script) - feel free to write me before December 21.

//Jim
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

I wanted to mention that a new release-0.13 has been pushed to SourceForge after some internal testing:
https://sourceforge.net/projects/vboxsvc/

Sources, an updated README and the recommended directory structure can be reviewed here: http://vboxsvc.svn.sourceforge.net/viewvc/vboxsvc/

Download as a flat tarball: http://sourceforge.net/projects/vboxsvc ... z/download
Download as a SysV package: http://sourceforge.net/projects/vboxsvc ... z/download

Updates in release 0.13 (2011-07-13):
* Added command-line parameters to more easily specify VM_NAME or SMF_FMRI URL when using the script "interactively"
* Moderately tested override of a timezone used to run the VM as compared to host TZ.
NOTE: If the VM is executed as a local user and if that user's profile somehow manages to override the TZ setting, it will be of higher priority than this method's override. User-profile's TZ will be used then.
** The KICKER hooks for an external got some real-life testing and cosmetic updates. They work indeed ;)

Feel free to give it a try and provide feedback! ;)
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Note: for internal reasons I am still stuck with VirtualBox 3.0.12 as the baseline version.
Testing and reports (and patches if need be) for managing recent VMs (i.e. VirtualBox 4.0.8) are encouraged. Help me help you! ;)
budy
Posts: 31
Joined: 1. Jul 2010, 17:51
Primary OS: OpenSolaris 11
VBox Version: OSE other
Guest OSses: MacOS, Linux

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by budy »

Hi Jim,

I have tried your SMF package on my Sol11 (151a) host and although the package seems to install fine, I cannot find it afterwards:

Code: Select all

root@sol11a:~/Software# pkgadd -d COSvboxsvc-0.13.pkg

The following packages are available:
  1  COSvboxsvc     SMF service for VirtualBox virtual machines
                    (intel) 0.13

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: 1

Processing package instance <COSvboxsvc> from </root/Software/COSvboxsvc-0.13.pkg>

SMF service for VirtualBox virtual machines(intel) 0.13
Center of Open Systems and High Technologies, MIPT, www.cos.ru; inspired by Alexandre Dumont's script
Using </> as the package base directory.
## Processing package information.
## Processing system information.
   9 package pathnames are already properly installed.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

Installing SMF service for VirtualBox virtual machines as <COSvboxsvc>

## Installing part 1 of 1.
/lib/svc/method/vbox.sh
/usr/share/doc/vboxsvc/README-vboxsvc.txt
/usr/share/doc/vboxsvc/TODO-vboxsvc.txt
/var/svc/manifest/site/vbox-svc.xml
[ verifying class <none> ]

Installation of <COSvboxsvc> was successful.
So, that looked okay, but:

Code: Select all

root@sol11a:~/Software# svcs -a | grep -i cos
root@sol11a:~/Software# 
I thought that the service is called COSvboxsvc, but svcs -a doesn't find it.

Cheers,
budy
jimklimov
Posts: 83
Joined: 7. Jul 2009, 08:28
Primary OS: OpenSolaris other
VBox Version: PUEL
Guest OSses: Linux, OSOL, Windows

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jimklimov »

Hello, budy

Did you happen to find the README file or the thread above, for that matter (this is where the README text originated)?

As the examples point out, each VM is managed by an individual service instance, usually named like "svc:/site/xvm/vbox:VM_NAME". "COSvboxsvc" is the package name in the manner long used in Solaris, with "COS" being the publisher and "vboxsvc" being a short informative name-tag of the software.

The easiest way to wrap your VMs into SMF service instances is to edit the service manifest XML file and clone the demo instance named "VM_NAME", and set whatever attributes you deem necessary. I hope this XML is self-explanatory (2/3 text is the comments ;) ). Then you just import the XML file and enable the new services:

# svccfg validate vbox-svc.xml
# svccfg import vbox-svc.xml
# svcs svc:/site/xvm/vbox:my-vm
STATE STIME FMRI
disabled 10:52:36 svc:/site/xvm/vbox:my-vm


Second usual approach is fully command-line, like this:

# svccfg -s vbox add illumos

# svcs -a | grep illumos
- svc:/site/xvm/vbox:illumos

# svcadm enable illumos

# svcs -a | grep illumos
online 2:08:10 svc:/site/xvm/vbox:illumos

# svccfg -s illumos listprop | grep vm/

After this you should "svccfg ... editprop" the VM settings to add the needed attributes.
For initial setup, if you already have some VMs running, the XML file is easier to craft. You can always export the running SMF service config into an XML file which includes all instances, edit it and import back with a new VM-instance :)
jamescollen
Posts: 1
Joined: 29. Sep 2011, 09:22
Primary OS: Linux other
VBox Version: OSE Debian
Guest OSses: Windows 7

Re: [Free as in beer] SMF service for VirtualBox VM's

Post by jamescollen »

Yeah, this is what I like about VirtualBox, they are really free as a Beer. And I love it being a regular beer drinker. Damn Love It!
A great collection of beer glasses, beer boot glasses!
Post Reply