Headless VirtualBox setup

VirtualBox, unless you look deeper, is a desktop application. However, it is possible to install it on a server without a monitor, and run virtual machines there. The setup procedure is somewhat quirky, and it has changed since the last time I did it. If you search for VBoxManage and VBoxHeadless, you’re likely to hit outdated instructions. If you’re reading this in 2020, these instructions are probably also out of date. Here’s the 2012 edition. Tested on Ubuntu Oneiric Ocelot.

Initial setup for installation of the OS from an ISO image. In my case, it was Solaris 10, which will be my small private porting and OpenCSW Solaris package building host.

UPDATE 2012-06-04: I’ve checked in the code into a subversion repository on code.google.com. The script is called vbox_setup.sh.

DISK_DIR="/path/where/you/have/space"
VM_NAME="Solaris 10 x86"
VDI="${DISK_DIR}/${VM_NAME}/Solaris-10.vdi"
CD="/path/to/sol-10-u9-ga-x86-dvd.iso"
DISK_SIZE=20000 # In MB
MEMORY_IN_MB=1600 # In MB

function setup {
VBoxManage createvm \
--name "${VM_NAME}" \
--basefolder "${DISK_DIR}" \
--register
VBoxManage modifyvm "${VM_NAME}" \
--memory "${MEMORY_IN_MB}" \
--acpi on \
--boot1 dvd \
--nic1 bridged \
--bridgeadapter1 eth0
VBoxManage createhd \
--filename "${VDI}" \
--size "${DISK_SIZE}"
VBoxManage storagectl "${VM_NAME}" \
--name "IDE Controller" --add ide
VBoxManage storagectl "${VM_NAME}" \
--name "SATA Controller" --add sata
VBoxManage storageattach "${VM_NAME}" \
--storagectl "SATA Controller" \
--type hdd --device 0 --port 0 \
--medium "${VDI}"
VBoxManage storageattach "${VM_NAME}" \
--storagectl "IDE Controller" \
--type dvddrive --device 0 --port 0 \
--medium "${CD}"

If something went wrong and you need to start over, you can nuke your VM with:

VBoxManage unregistervm "${VM_NAME}"
rm -rf "${DISK_DIR}/${VM_NAME}"

To start your virtual machine:

VBoxHeadless -s "${VM_NAME}" --vnc

You can now connect to your virtual machine via VNC. You will probably need to set up a VNC tunnel and use a VNC viewer. Ubuntu has a remote desktop application.

When you’ve installed your guest OS, you need to remove the installation media, otherwise your VM will start the installation process all over again. You can stop the VM (I just press CTRL+C, there must be a better way).

VBoxManage storageattach "${VM_NAME}" \
--storagectl "IDE Controller" \
--type dvddrive --device 0 --port 0 \
--medium none

You can start your VM again, this time it will boot into the installed OS.

FLOSS Weekly 163: OpenCSW, addendum

FLOSS Weekly, a podcast about Free-Libre and Open Source software, episode 163 featured OpenCSW, a project I actively participate in.

Since I was not on the podcast, I would like to use this opportunity to add to what has been said there.

Q: 05:30 What is OpenCSW and what does it contribute to the world?
A: …to add to what Phil said (we provide packages free as in free beer), there are two parts of what we provide: one part is binary packages, and the other part is the source code to build these packages.  It hasn’t been historically the culture at OpenCSW (or formerly, Blastwave) to release build recipes.  At OpenCSW, the policy for all new maintainers is to release source code of all packages they build.  However, there is still a number of old-timers, who build packages using own, unpublished scripts.  We are making efforts to have all build recipes published as open source, and while we’re still not there yet, it’s one of the most important points on our agenda.  In this sense, we do care about freedom and about being an open source project.

Q: 15:15 Do you think of OpenCSW as of a Solaris distribution?
A: Yes, as much as it is possible, while being based on commercial Solaris. The main difference between OpenCSW and Linux or BSD distributions is that OpenCSW does not provide the base OS, such as the kernel, libc or an installer.  From the perspective of a business which runs third party applications, it’s important that their OS is supported by the vendor.  Nexenta is a lovely Debian-based system with a Solaris kernel, but you can’t get support for an Oracle database on it.

How do I install a Solaris package?

Shortly after joining a new team at work, I learned that team supports quite a number of machines running Solaris. While talking to the team about how the maintenance is done, the following question came up:

“How do I install packages?”
“You go to the server that keeps packages, you use ssh to copy the package in the directory format to the target machine, then you log into the target machine, and you run pkgadd.”
“No no, that’s not what I was asking. If I need to install a package on our servers, what do I do?”
“You use ssh to copy the package in directory format…”
“No no, not just one package on one server, I meant installing many packages on many servers.”
“You take the directory format package…”

It looked like I had an idea for a project: create a workable package management solution.  I ended up completing that project, which resulted in me being involved in the OpenCSW community. The general idea was like this:

• There’s an internal mirror of the OpenCSW package archive.
• An internal staging is implemented: we don’t install directly from the mirrored directory. Instead, copies called ‘unstable’, ‘testing’ and ‘stable’, are made.  Servers pull packages from ‘stable’ by default.  A chosen subset pulls packages from ‘unstable’ and ‘testing’.
• There is an internal package overlay, containing additional, in-house built packages.  A wrapper around bldcat is used to create this repository. The same staging approach is used.
• puppet is used to deploy packages.  pkgutil provider is responsible for interacting with pkgutil, which in turn is responsible for interacting with the Solaris pkgadd utility.  pkgutil is able to download packages from multiple package repositories.

If you want to know more or talk to project members, visit #opencsw on Freenode.

System-wide GAR

If you’re building Solaris packages using GAR from OpenCSW, you’re familiar with the GAR code inclusion practice. Each package build recipe can be checked out individually, and pulls GAR code using the svn:externals parameter. Try the following command if you don’t know what I mean:

svn co https://gar.svn.sourceforge.net/svnroot/gar/csw/mgar/pkg/mysql5

This sort of works, but since SourceForge Subversion repository works somewhat slow, it can be irritating if there’s a GAR update you want to roll out to all your checked out packages.

As an alternative to svn:externals, you can use a system wide GAR installation, and use a symlink:

ln -s /opt/csw/src/gar/v2 gar

There’s currently no official GAR package, but you can build it yourself from its Subversion build recipe.

There’s one problem. If you don’t have the symlink and your Makefile tries to include the gar/category.mk file, your command will fail with:

Makefile:29: gar/category.mk: No such file or directory
gmake: *** No rule to make target `gar/category.mk'.  Stop.

It would be nice to make the symlink if it didn’t exist, right? I’ve found a way to do it:

ifeq ($(shell test -h gar && echo yes || echo no), yes)
include gar/category.mk
else
%:
        @echo "Making the symlink to GAR"
        ln -s /opt/csw/src/gar/v2 gar
        @echo "Please re-run your command."                     
endif

The idea is that it only includes the file if the symlink exists. Otherwise, it makes the symlink and asks us to re-run the gmake command.

I haven’t so far found a way to make it work in one go.

SUNW packages in an OpenCSW-format catalog

OpenCSW provides a software stack which allows to build catalogs with Solaris packages, and use a utility, such as pkg-get or pkgutil to download and install packages together with all the dependencies.  It works very nice for OpenCSW packages, but what if one wants to install a SUNW, that is, a Sun-provided package?  Solaris DVDs are not in the OpenCSW format and pkgutil can’t use them directly.

I wrote a tool which converts a collection of Sun packages in the directory format (they’re distributed this way) into a set of gzipped srv4 format files (.pkg.gz) which can be served over http or ftp and downloaded via pkgutil.

To get started:

1. Install pkgutil and pkgutilplus from OpenCSW
2. Check out the OpenCSW utilities:
   svn co https://opencsw.svn.sourceforge.net/svnroot/opencsw/utilities/
3. Have a Solaris DVD available in your filesystem
4. Patch /opt/csw/bin/bldcat if necessary (the patch is there at the end of this post)
5. Run the package converter:
   ./build_sun_catalog.py -p “/somewhere/Solaris_10/Product” -c “/target/directory/sparc/5.10”
6. Run bldcat and create the catalog:
   (cd /target/directory/sparc/5.10; bldcat .)
7. Server the target directory via http under, say http://packages.example.com/solaris-10u5
8. From your Solaris machine, use the following command to install a package together with all the dependencies:
   pkgutil -t http://packages.example.com/solaris-10u5 -i SUNWfoo

A word of warning though: upgrading SUNW packages can break your system! If you attempt to upgrade a core Solaris package such as SUNWcakr or SUNWkvm.  If pkgutil tries to do that, use the -x option to filter out those packages. For example:

pkgutil -t http://packages.example.com/solaris-10u5 -i SUNWfoo -x SUNWcakr -x SUNWcar -x SUNWkvm

--- bldcat.orig 2009-11-23 13:45:58.436220417 +0000
+++ bldcat      2009-11-23 13:42:27.098746854 +0000
@@ -100,3 +100,3 @@
foreach (glob "$tmpdir/*") {
-    ($pkgname) = (/([\w\+-]+)$/) if (-d);
+    ($pkgname) = (/([\w\+-\.]+)$/) if (-d);
}

OpenCSW is where the goodness is

Executive summary: OpenCSW rocks. http://ftp.heanet.ie/pub/opencsw/ (and the like) is the mirror setting you want. (See the README files for more mirrors.)

Solaris 10, as installed from a DVD, has a very small number of software packages. It has packages that are relevant to the OS itself, but it does not have packages with things such as Apache or PostgreSQL vim, or subversion client, or GNU utilities (awk, grep, etc). These ones you need to either compile yourself or use a precompiled binary package. A project called blastwave used to provide those for quite a number of years. However, sometime in August 2008, “things happened”. What those things are, I’ve decided I don’t want to know. The current state of affairs is that there are 2 (two) projects now, one is called “blastwave” and one is “opencsw”. I did a bit of research on the web to find out about their source repositories and binary package statuses.

One thing that struck me pretty strong is that none of those projects advertises its code repository. With open source projects, the code repository is usually the first thing published. “We have a new project! Here’s our code: http://svn.example.org!” That’s what the phrase “open source” means, according to a common sense understanding. If you’re interested in the project, you can download the source, tinker with it and if you come up with something useful, you offer your change in a form of a patch. Without the source code, you can’t do that. So, where’s the source?

Finding the source code of OpenCSW was a bit harder than the source code of Blastwave. Here it is:

http://gar.svn.sourceforge.net/viewvc/gar/csw/mgar/pkg/

To check out the source (takes a long time to complete):

svn co --ignore-externals https://gar.svn.sourceforge.net/svnroot/gar/csw/mgar opencsw

The code repository, looking at the statistics page, looks pretty active. There’s also a wiki page with instructions.

If you want to update a package or change something in a package from CSW, or you’re wondering how does package building work there, check out this source, then change into a directory with a package, like cups/trunk and type ‘gmake’. It works in a similar fashion to BSD ports.

I didn’t do full exhaustive stats, but it seems that packages in OpenCSW are generally much more recent than the ones from Blastwave. So, check your pkg-get.conf (yes, it’s a newer version of or pkgutil.conf if you use this alternative to pkg-get) and replace the old blastwave mirror with OpenCSW one, and see the goodness flowing in.

(updated on 2009-03-11 according to Phil’s comments)