[old] Unattended Guest OS Install - vbox-unattended

Discussions related to using the OSE version of VirtualBox.

[old] Unattended Guest OS Install - vbox-unattended

Postby Technologov » 19. Dec 2011, 15:38

(a.k.a. coffee break system installation)

This is the implementation. For Research Paper, see this:
https://forums.virtualbox.org/viewtopic ... 75#p211275

Install instructions: (Debian Linux)
1. New Dependencies:
# apt-get install p7zip-full mtools genisoimage
2. Download & install patched VirtualBox-unattended-OSE for Linux: (4.1.29)
http://www.qumble.org/Downloads/Virtual ... r89849.run
2.a. [OPTIONAL] MD5SUM:
2e22e1568a49c0585ab7d4dee6eec50e /home/alexey/1/VirtualBox-4.1.29_unattended-OSE-r89849.run
3. Make sure you use Oracle's guest additions:
Download http://download.virtualbox.org/virtualb ... 4.1.28.iso
cp [path]/VBoxGuestAdditions_4.1.28.iso /opt/VirtualBox/additions/VBoxGuestAdditions.iso

Good news !
I have finally achieved first milestone - first working version !
This project lets you to install any supported guest OS in 10 minutes !

Supported Guest OSes:
------------------------------------
1. Windows NT 5 (tested with Windows XP, 32bit)
2. Windows NT 6 (tested with Windows Vista, 32bit)
3. Red Hat kickstart (tested with RHEL 3/4/5, 32+64bit)
4. Debian preseed (tested with Debian 6.0 DVD, both 32+64bit)
5. SUSE autoyast (tested with openSUSE 11.3/11.4, 32+64bit)
------------------------------------

Following the topics:
https://forums.virtualbox.org/viewtopic.php?f=9&t=26005
and
https://forums.virtualbox.org/viewtopic.php?f=9&t=42926

I decided to write a small program, that does exactly this work:
named 'vbox-unattended'

Why I write 'vbox-unattended' ?

Because:
Installing Windows XP by coffee-drinking people takes about 30 min
Installing Windows XP by active geek, clicking instantly takes about 15 min
Installing Windows XP by Unattended Install, takes about 10 min, letting people to drink coffee in first place. :)
(on standard Core2-class hardware with HDD)

Also because:
*no reason to let VMware to steal whole market (but that's secondary reason)

-----------------------------------------------------------
Design Goals:
-----------------------------------------------------------
The system should:
1. run on any host OS supported by VirtualBox, (I try to avoid host-OS-specific tricks, if possible, but for now only Linux + FBSD host is implemented)
2. be simple to implement,
3. be offline (i.e. not require an Internet connection),
4. be fast (i.e. generate only minimal remastered boot CD image, rather than whole OS CD image)
5. support both Windows guests and several major Linux guest families (Debian, Red Hat, SUSE).
6. be Free Software (dependency on non-free software is forbidden, support is 'OK')

Target: Desktop users. (but also useful for: sysadmins, VBox QA, distro-hoppers, newbies)
-----------------------------------------------------------
vbox-unattended was written on Debian GNU/Linux 6.0 and tested with VirtualBox 4.1.
It currently supports only Linux-hosts with Windows guests (NT 5x and NT 6x families: Windows 2000/XP/2003/Vista/7/2008 guests, tested with WinXP/Vista, 32-bit), Red-Hat-like Linux guests. (Tested with RHEL3/4/5 32-bit)), Debian guests (tested with Debian 6), and SUSE guests (tested with openSUSE 11.3/11.4, 32-bit).
Due to it's beta quality the software may not function properly as advertised.
Because the basic design is OS-neutral, porting it from Linux to Windows hosts should be possible later...

TODO:
-more tests and stability
-pushing upstream

How to use: (python version)
Python version works like a 3rd party application, with standard VirtualBox v4.1.x.
1. edit vboxunattended.py - last 10 lines - this will configure your stuff.
2. run it:
$ python ./vboxunattended.py

Release Notes:
1. It works !
2. Full VirtualBox GUI and Main API integration ! (for C++ version)
3. To install Debian 6 guest, you need the first DVD, 1 of 8. (32-bit or 64-bit Squeeze)
4. Linux / FreeBSD hosts only
5. working from physical CD-ROM is not supported. (only with ISO images)
6. for each run, you must change VM name inside "vboxunattended.py" - in last 10 lines. (python version only)
7. RHEL6 and Fedora guest OS support are broken.
8. there is no cleanup code, so you can study the leftovers... (@_@)

Changelog:
-v0.1 - 2011-12-19: First release. supports Linux hosts and Windows NT 5.x and NT 6.x guests (python only)
-v0.2 - 2011-12-20: added support for Red-Hat-like Linux guests (via Red Hat Anaconda kickstart backend)
-v0.3 - 2011-12-24: added support for VirtualBox Guest Additions + improved stability.
-v0.4 - 2011-12-29: added support for Debian Linux guests (via Debian preseed backend)
-v0.5 - 2011-12-31: added support for SUSE Linux guests (via AutoYast backend) + improved code arch.
HAPPY NEW YEAR 2012 !
v0.5.1-2012-01-01: hugely improved support for Debian guests.
v0.6.0-2012-01-10: Complete rewrite in C++ ! Features VirtualBox GUI integration !
v0.6.1-2012-01-12: First BETA ! Basic functionality is mostly stable. C++ and python combined release !
v0.6.2-2012-01-17: Fixed bug with folders with spaces (hosts) and workaround for Debian Bug #655841. (Debian guests)
v0.6.3-2012-01-17: added support for FreeBSD hosts and small fix for Debian guests. (thanks to "nox-")
v0.7 - 2012-02-01: Near complete rewrite in COM C++, integrated into VBox Main API.
v0.7.1 - 2013-10-18: Refactored patch to apply on top of VirtualBox OSE 4.1.28. + First binary build that you can download and run !

v0.6.1. details:
1. Combined release in C++ and python. (they have similar function names,
and similar features). GUI is provided with the C++ version.
2. Critical bug fixed: libunattended.cpp in file read code. (C++ only)
3. Red Hat guest support went throught a major rewrite:
- added support for Red Hat Enterprise Linux guests (C++ version)
- added "password encrypter" (this solves the problem, that
Red Hat requires encrypted passwords by crypt(3) POSIX function.
- added better granularity to IGuestOSType (C++ only)
The bad news: Introduction of RHEL guests forced me to make incompatible changes in VirtualBox.
The v0.6.1 C++ version is not 100% compatible with the original.
4. As per <Teknomancer>'s recommendation, I replaced system("mkdir")
with "IPRT RTDirCreate()" (C++ only)
5. Debian 6.0 guest found to be non-deterministic, swapping
between /dev/cdrom and /dev/cdrom1 at random.
Problem hopefully solved by using /dev/sr0, which seems reliable.
6. python code got much better abstraction, and much code removed.

v0.7:
vbox-unattended has now integrated much deeper into the core of VirtualBox.
+Introduces New Main API: IMachine::IUnattended
+customizable Unattended pipeline. You can hook into the Unattended pipeline and change things on the fly.
+3rd near complete rewrite. (1st was in python, 2nd in Qt C++)
+GUI: First Run Wizard is now disabled for unattended machines.
+Visually and functionally it is a carbon copy of v0.6.3.

License: MIT/X11.
====
-Technologov
Attachments
vbox-unattended-0.7.1.patch.txt
Re-factored patch to apply on VirtualBox 4.1.28 OSE. + first binary build !
(124.61 KiB) Downloaded 135 times
vbox-unattended-v0.7.0-beta.patch.tgz
Near Complete rewrite in COM C++ !
(26.8 KiB) Downloaded 204 times
vbox-unattended-v0.6.3-beta.tar.gz
(15.14 KiB) Downloaded 152 times
vbox-unattended-v0.6.2-beta.tar.gz
(15.08 KiB) Downloaded 62 times
vbox-unattended-v0.6.1-beta-src.zip
Combined C++ and python release ! First BETA !
(35.51 KiB) Downloaded 99 times
vbox-unattended-v0.6-alpha.patch.zip
Complete rewrite in C++ ! NOTE: v0.6 is less stable than v0.5, which is in python.
(12.31 KiB) Downloaded 56 times
vbox-unattended-v0.5.1-alpha-src.zip
Last pure python release.
(23.42 KiB) Downloaded 100 times
vbox-unattended-v0.5-alpha-src.zip
(22.15 KiB) Downloaded 51 times
vbox-unattended-v0.4-alpha-src.zip
(22.93 KiB) Downloaded 65 times
vbox-unattended-v0.3-alpha-src.zip
(19.7 KiB) Downloaded 52 times
vbox-unattended-v0.2-alpha.zip
source code
(15.72 KiB) Downloaded 55 times
vbox-unattended-v0.1-alpha.zip
source code
(12.21 KiB) Downloaded 71 times
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 19. Dec 2011, 16:52

New GUI design: (C++ version)

The standard 'NEW VM' GUI will be modified at step #2 to add 'unattended' option. (added on 07.Jan.2012)
vbox-new-vm-dialog2-modified.png
v1.0
vbox-new-vm-dialog2-modified.png (63.61 KiB) Viewed 15927 times


If the new 'Unattended Install' option was chosen, then new GUI wizard will take place, that asks for 'Unattended' specifics.
In addition the CD-ROM media is moved into this step (instead of 'First Run' Wizard)

vbox-new-dialog.png
v2.0
vbox-new-dialog.png (58 KiB) Viewed 15894 times


Ideas / Q&A:
Q: Why I think it should be part of 'New VM' Wizard ?
A: Because it seems to fit best into this class code-wise.

Q: Why 'Unattended Wizard' cannot be part of 'First Time Wizard' ?
A: Reason: The unattended scripts are pretty hard-coded and cannot support endless number of possible virtual hardware configurations, that the user might configure in 'VM Settings', before starting the VM. IMO it is impossible to write unattended scripts to support unknown hardware reliably (SATA vs. IDE vs. SATA; Primary Master vs. Primary Slave, etc...).
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby axlrod » 4. Jan 2012, 17:20

I just sat and read trough all your code and I just needed to reply to this and say...

WAUV, lots of hours in this. I'm sure this will come in handy for people and it looks very easy to customize to your own setup.

Thumbs up from here.
axlrod
 
Posts: 5
Joined: 8. Jun 2011, 00:14
Primary OS: Mac OS X other
VBox Version: OSE Debian
Guest OSses: Windows 10

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 6. Jan 2012, 04:57

yea, thanks.

Now the big problem is future direction of this project:
Two options:
A. keep 'vbox-unattended' as-is - separate project from VirtualBox core.
For me, it would be sad, because newbies (whom gain the most) won't be able to find it.
I can port it to Windows and write GUI in this case. Windows version will be *very* ugly -- it will weight > 100 MB due to huge dependencies. (python+pyQt+Java)
Linux version will weight around 30 KB (kilobytes !)

B. somehow attempt to integrate this into VirtualBox proper.
I lack the technical skill to make quality integration, although ugly patches are possible.
Quality integration requires good skills in C/C++, which I lack, plus good GUI design.
Also a big question is GUI design -- it is hard to design is useful GUI that will integrate with my new engine and fit into current VBox design.
In this case porting to Windows host and separate GUI need not to be written.
Quality integration of my technology into VirtualBox will increase it's size, by, perhaps 100-200 KB. It is that small.

I have not decided yet as to which route is preferable.

Jeremy Grossman wrote: leader of GNS3 project wrote:
<grossmj> the GUI part looks good
<grossmj> I would prefer an integration to vbox, option B
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Perryg » 9. Jan 2012, 20:52

Couple of questions.
    1) what about Solaris and MAC? I doubt this will be included in the main trunk if it does not support at least Solaris, but that is just my 2cents.
    2) error control, IE. grayed out if the GUI detects that you are installing a non-supported guest. Or when you input a wrong string?
    3) how are you doing regression testing? older versions that work but are not totally supported.
    4) deb and rpm should be able to understand the depends and install the missing ones, have you taken that into account?
    5) any idea how much more weight this is going to add to the DL/install files or the total installed package?
Perryg
Site Moderator
 
Posts: 34373
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 9. Jan 2012, 21:50

1) what about Solaris and MAC? I doubt this will be included in the main trunk if it does not support at least Solaris, but that is just my 2cents.


Windows hosts:
I need some cross-platform FAT12 tools for them; on Linux I use 'mtools'.
Windows hosts lack free, small, native tools to do this job. (I know 2 tools, one is dependent on cygwin, other on Java, so both are ugly, and won't be accepted by Oracle)
The recommended solution, is to port Qemu's FAT12 code to VirtualBox, which will allow us to support all host OSes, and solve at fundamental level. Some experienced C developer will have to do this task. It is MIT-licensed, so it's possible.

For now, the plan is to enable this feature only on Linux hosts, and see how someone solve this later. (hopefully for v4.2.0)
My feature *can* work on Windows hosts without the FAT12 code, but will be limited to Red Hat/SUSE guests. Windows and Debian guests will not work.

Solaris/Mac/BSD hosts: Someone has to research the option to use 'mtools' there, and if not, then wait for someone to port Qemu's code over here.

Mac/BSD guests:
I don't know if automating those is possible. Never researched.

Solaris guests:
Solaris provides "Jump Start" technology, so writing a backend is possible. It is similar in concept to Red Hat's "Kick Start", but different in detail.
I have no plans to write such thing, however.

2) error control, IE. grayed out if the GUI detects that you are installing a non-supported guest. Or when you input a wrong string?


Yes, I will do _some_ error control, based on Guest OS Type.
There is no way to do _complete_ control, because the Guest OS Type is not granular enough: i.e. you can specify "Fedora", but not "Fedora 7", etc...
In general, the situation will be much like with VBox Linux Additions. (i.e. it will mostly work, but not guaranteed to work on ancient versions of Linux guest OSes, or on too new versions). In reality it will work well on stable OSes, such as RHEL and Debian, and will work poorly for fast-changing OSes, such as Fedora.
I plan to test certain Linux guests, and they will be fully supported. The test results will be published.

With Windows guests, the quality control will be 100%, because VBox provides granularity (correct GuestOS Type) for Windows guests.
I plan to support everything from Windows 2000 and up.

3) how are you doing regression testing? older versions that work but are not totally supported.


The same way, as I test Guest Additions, by using my stability matrix.

4) deb and rpm should be able to understand the depends and install the missing ones, have you taken that into account?


ofc.
I take advantage of automatic dependency resolver.

5) any idea how much more weight this is going to add to the DL/install files or the total installed package?


If done properly, it will add about 100 kilobytes of code. (not much vs. the 300+ MB of code VBox already has...)

+new dependencies:
  • 7-zip
  • genisoimage
  • geteltorito
  • mtools -or- Qemu's FAT12 code porting.

The new dependencies will make the Windows version more bloated by few MB, while Linux version will stay the same. (because Linux OS has those)

-Technologov
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Perryg » 9. Jan 2012, 22:01

OK now I am totally confused.
My feature *can* work on Windows hosts without the FAT12 code, but will be limited to Red Hat/SUSE guests. Windows and Debian guests will not work.

Is this a private project or is it something that you are working on for the Oracle/VirtualBox project? As it stands this will never make it into the trunk and from the looks of it you probably will never make it in unless you know C++, The DEVs are too busy to handle this project from what I understand.

Just for my edification when you say we, who are you talking about?
Perryg
Site Moderator
 
Posts: 34373
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 9. Jan 2012, 22:10

>Is this a private project or is it something that you are working on for the Oracle/VirtualBox project?
private project.

>Just for my edification when you say we, who are you talking about?
"me", no "we".
"we" as in "community", if someone wishes, to... If not, then Linux hosts only.

I have clarified this a bit, in my original text.

-Technologov
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Perryg » 9. Jan 2012, 22:18

OK Thanks.
Perryg
Site Moderator
 
Posts: 34373
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 9. Jan 2012, 22:19

>you probably will never make it in unless you know C++,

I already have some C++ code, and I think I will _make_ it working in few weeks.
I write 2 compoents:
1. libunattended.cpp - generic unattended library in C++, with few specifics towards VirtualBox (only Guest Additions); This is similar to my current libunattended.py library, so the core logic is tested.
2. I have modified the VirtualBox GUI code already.

So the next step is to glue the two components together, and test the whole.
_I_ believe, that I will have a working C++ solution in few weeks. Linux hosts only.

-Technologov
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 10. Jan 2012, 18:12

update: I have decided to make it slightly earlier, and released the C++ version _today_.

v0.6. patch against VirtualBox.

First experimental patch of my new lovely feature, now re-written in C++ !

Release Notes: (v0.6)
1. It works ! (tested with 32-bit guests: Windows XP, Vista, openSUSE
11.4, Debian 6)
2. Full VirtualBox GUI integration !
3. you must symlink all template* files to /tmp/ (I have not found a
way retrieve vbox locations properly)
3.a. symlink GA.iso into "/tmp/VBoxGuestAdditions.iso" (for same reason)
4. Linux hosts only
5. Red Hat guests not supported in this release.
6. patch must be applied vs. vbox-svn-2012-01-07
7. working from physical CD-ROM is not supported. (only with ISO images)

Dependencies:
7z (7-zip) - used to extract data from ISOs.
mtools - used to copy data to floppy images.
genisoimage & geteltorito

It would be nice to stabilize it, and include into 4.2.0.

License: core logic under BSD, all my vbox-patches under MIT. (I can
change if needed)

Happy Hacking !
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 23. Jan 2012, 00:20

Main API: IMachine::IUnattended:

Currently I am aware of 5 projects trying to achieve Unattended installs with VirtualBox, to varying degrees (openSUSE openQA, Technologov's vbox-unattended, JClouds, pallet, vagrant-VeeWee)
In order to remove duplication of effort, it is a good idea to implement this at the Main API level.

/* Attributes: */
BSTR user (R/W) <-- create a new user, named <user>.
BSTR password (R/W) <-- password is both for normal user and for Administrator / 'root' accounts.
BSTR ProductKey (R/W) <-- must have notation: "12345-12345-12345-12345-12345". Windows guests only.
BSTR ISOpath (R/W) <-- a full path to original OS ISO.
BOOL InstallGuestAdditions (R/W) <-- add an option to enable / disable automated VirtualBox additions install.

/* Functions: */
/**
* Generates Guest-OS-specific unattended scripts (depending on IMachine::OSTypId),
* and puts them into the MachineFolder/...
*
* The parameters are replaced according to IUnattended attributes.
*
* The API user can edit, customize or completely replace those scripts,
* before final packaging.
* The unattended scripts are text files, that have Guest-OS-specific format.
*
* What gets generated ?
*
* Windows NT 5.x:
* MachineFolder/winnt.sif
* Windows NT 6.x:
* MachineFolder/autounattend.xml
* Red Hat:
* MachineFolder/ks.cfg
* MachineFolder/isolinux.cfg
* Debian:
* MachineFolder/preseed.cfg
* MachineFolder/isolinux.cfg
* SUSE:
* MachineFolder/autoinst.xml
* MachineFolder/isolinux.cfg
*/
constructUnattendedScript() <-- will edit scripts automatically, and put them into correct places ( MachineFolder/... )

/**
* This function finalizes the task of the Unattended pipeline.
* Packages Guest-OS-specific unattended scripts into CD bootable ISOs,
* and into floppies, depending on IMachine::OSTypId.
*
* Windows guests automatically read their unattended config files from
* a floppy disk, and so do not require custom bootable medium.
*
* Linux guests:
* Because Linux guests do not automatically read their unattended scripts,
* We must generate bootable CD on the fly for them.
* In order to make CDs bootable, we first extract boot files from Linux OSes,
* namely: Linux kernel, initrd, isolinux bootloader. Read: [boot contents]
* Plus we extract LinuxAdditions from the Guest Additions ISO, and
* also package this together on the bootable ISO.
*
* What gets generated ?
*
* Windows NT 5.x:
* MachineFolder/floppy_script.img
* Windows NT 6.x:
* MachineFolder/floppy_script.img
* Red Hat:
* MachineFolder/redhat_boot.iso
* MachineFolder/bootiso/[boot contents]
* Debian:
* MachineFolder/debian_boot.iso
* MachineFolder/bootiso/[boot contents]
* MachineFolder/floppy_script.img
* SUSE:
* MachineFolder/suse_boot.iso
* MachineFolder/bootiso/[boot contents]
*/
packageUnattendedScript() <-- will copy into floppy / CD images, and put them into correct places ( MachineFolder/... )

NOTE: between those 2 stages, it is possible for the API user to edit those files, either manually (in case of VBoxManage CLI user) or via stream editor, such as "sed". This will give us pretty customizable solution.

Status: DONE in v0.7
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 1. Feb 2012, 20:30

Work Complete. v0.7 released !
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 18. Oct 2013, 21:03

First binary release !!! VBox-unattended-OSE 4.1.29 (patch 0.7.1)
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Re: Unattended Guest OS Install - vbox-unattended

Postby Technologov » 6. Dec 2014, 18:59

Finally ! The unattended patch from VirtualBox v4.1.x (patch 0.7.x) was ported to VirtualBox 4.3.x (patch 0.8.x) !
It was a major rewrite, and it is less stable than 0.7.1.

vbox-unattended-0.8-alpha1.patch - Dec.2014
Attachments
vbox-unattended-0.8-alpha1.patch.txt
(181.58 KiB) Downloaded 37 times
Technologov
Volunteer
 
Posts: 3313
Joined: 10. May 2007, 16:59
Location: Israel

Next

Return to VirtualBox OSE

Who is online

Users browsing this forum: No registered users and 2 guests