Install using vmbuilder on Ubuntu

vmbuilder is a command line tool for creating virtual machines on Ubuntu. It can only create Ubuntu installations, contrary to virt-manager and the collection of tools that use libvirt. This is one way to use scripts instead of GUI for installation, although virt-manager can be used after for additional support.

The purpose of this document is to describe how to easily get a working Foswiki site by using vmbuilder. I have not found any duplicates of this effort and hope this is the right place for this document. Vmbuilder is quite well documented so this document is more to try to make a complete and easy process for getting a new server and Foswiki site up and running, ready for test or whatever you desire to do with it.

wip This is under construction. Anyone is invited to join. Comments at the bottom.

Testing is done on Utuntu 10.04, 10.10 and 11.04. Help wanted testing. There are unofficial deb files and repository for Debian based distros if that would be preferable.

This process generates a new virtual machine in a few minutes. On my laptop it takes about 3 minutes. And it does not require CD-ROM, ISO's or other stuff than your existing Ubuntu installation.

The vm's generated can be managed with the virt-manager GUI, added automatically. The vm will be a clean current stable Foswiki and dedicated to that task. It will be a Ubuntu LTS server and you can of course do whatever modifications you wish to both the install described here and after the server is created. It will be a normal Ubuntu server.

Hopefully this will end up with at least two bash scripts, one for setting up the build machine, your local Ubuntu install, and another for quickly getting a ready Foswiki site ready.

Build machine

You need to install KVM and some packages to get started. Here's a sample command for a typical setup:
apt-get install kvm qemu-kvm bridge-utils build-essential virtinst virt-manager apt-cacher subversion

See vmbuilder and JeOS.

Add yourself to the group libvirtd. (Log out and back in necessary?).
usermod $(whoami) -a -G libvirtd

Basics of vmbuilder

Sample command. The \ escapes newline so this can be copy/pasted into a bash command prompt:
vmbuilder kvm ubuntu \
--mem=1024 \
--cpus=1 \
--hostname=vm01 \
--dest=vm01 \
--mirror=http://192.168.122.1:3142/ubuntu \
--user=wikiadmin \
--name=administrator \
--pass=secret \
--rootpass=verysecret \
--libvirt=qemu:///system \
--rootsize=2048 \
--swapsize=512 \
--overwrite \
--tmpfs -

When the command has run successfully start the vm via virt-manager or virsh --connect qemu:///system start vm01

Remember to start apt-cacher first if you want caching:
#apt-cacher
Run in background or have it autostart if you need. I only start it manually when needed.

The DEST parameter will create a directory with given name and put files there. Disk files will default go to /var/lib/libvirt/images or /etc/libvirt/ and also the config file used for virt-manager.

Virtual machines can be for KVM, VMWare (vmw6, vmserver), Xen. For now this is focused on KVM and adding to virt-manager. If you want to skip virt-manager and not use any GUI, just skip that parameter and vmbuilder will create a file called run.sh in the directory with the virtual machine. In that file you see how to start the vm with just the disk file and parameters.

KVM does not need the xml config file, you can start a virtual machine by specifying the disk file and appropriate parameters.

#vmbuilder kvm ubuntu --help
Usage: vmbuilder hypervisor distro [options]

Options:
  -h, --help            show this help message and exit
  --version             Show version information

  Build options:
    --debug             Show debug information
    -v, --verbose       Show progress information
    -q, --quiet         Silent operation
    -o, --overwrite     Configuration file
    -c CONFIG, --config=CONFIG
                        Configuration file
    --templates=DIR     Prepend DIR to template search path.
    -d DESTDIR, --destdir=DESTDIR
                        Destination directory
    --only-chroot       Only build the chroot. Don't install it on disk images
                        or anything.
    --chroot-dir=CHROOT_DIR
                        Build the chroot in directory.
    --existing-chroot=EXISTING_CHROOT
                        Use existing chroot.
    -t DIR, --tmp=DIR   Use TMP as temporary working space for image
                        generation. Defaults to $TMPDIR if it is defined or
                        /tmp otherwise. [default: /tmp]
    --tmpfs=SIZE        Use a tmpfs as the working directory, specifying its
                        size or "-" to use tmpfs default (suid,dev,size=1G).

  Disk:
    --rootsize=SIZE     Size (in MB) of the root filesystem [default: 4096]
    --optsize=SIZE      Size (in MB) of the /opt filesystem. If not set, no
                        /opt filesystem will be added.
    --swapsize=SIZE     Size (in MB) of the swap partition [default: 1024]
    --raw=PATH          Specify a file (or block device) to use as first disk
                        image (can be specified multiple times).
    --part=PATH         Specify a partition table in PATH. Each line of
                        partfile should specify (root first):      mountpoint
                        size  one per line, separated by space, where size is
                        in megabytes. You can have up to 4 virtual disks, a
                        new disk starts on a line containing only '---'. ie:
                        root 2000      /boot 512      swap 1000      ---
                        /var 8000      /var/log 2000

  Other options:
    --ssh-key=PATH      Add PATH to root's ~/.ssh/authorized_keys (WARNING:
                        this has strong security implications).
    --ssh-user-key=SSH_USER_KEY
                        Add PATH to the user's ~/.ssh/authorized_keys.
    --manifest=PATH     If passed, a manifest will be written to PATH

  Post install actions:
    --copy=FILE         Read 'source dest' lines from FILE, copying source
                        files from host to dest in the guest's file system.
    --execscript=SCRIPT, --exec=SCRIPT
                        Run SCRIPT after distro installation finishes. Script
                        will be called with the guest's chroot as first
                        argument, so you can use 'chroot $1 <cmd>' to run code
                        in the virtual machine.

  Settings for the initial user:
    --user=USER         Username of initial user [default: ubuntu]
    --name=NAME         Full name of initial user [default: Ubuntu]
    --pass=PASS         Password of initial user [default: ubuntu]
    --rootpass=ROOTPASS
                        Initial root password (WARNING: this has strong
                        security implications).
    --uid=UID           Initial UID value.
    --gid=GID           Initial GID value.
    --lock-user         Lock the initial user [default: none]

  Package options:
    --addpkg=PKG        Install PKG into the guest (can be specified multiple
                        times).
    --removepkg=PKG     Remove PKG from the guest (can be specified multiple
                        times)
    --seedfile=SEEDFILE
                        Seed the debconf database with the contents of this
                        seed file before installing packages

  General OS options:
    -a ARCH, --arch=ARCH
                        Specify the target architecture.  Valid options: amd64
                        i386 lpia (defaults to host arch)
    --hostname=HOSTNAME
                        Set NAME as the hostname of the guest. Default:
                        ubuntu. Also uses this name as the VM name.

  Scripts:
    --firstboot=PATH    Specify a script that will be copied into the guest
                        and executed the first time the machine boots.  This
                        script must not be interactive.
    --firstlogin=PATH   Specify a script that will be copied into the guest
                        and will be executed the first time the user logs in.
                        This script can be interactive.

  Installation options:
    --suite=SUITE       Suite to install. Valid options: dapper gutsy hardy
                        intrepid jaunty karmic lucid maverick natty [default:
                        lucid]
    --flavour=FLAVOUR, --kernel-flavour=FLAVOUR
                        Kernel flavour to use. Default and valid options
                        depend on architecture and suite
    --variant=VARIANT   Passed to debootstrap --variant flag; use minbase,
                        buildd, or fakechroot.
    --iso=PATH          Use an iso image as the source for installation of
                        file. Full path to the iso must be provided. If
                        --mirror is also provided, it will be used in the
                        final sources.list of the vm.  This requires suite and
                        kernel parameter to match what is available on the
                        iso, obviously.
    --mirror=URL        Use Ubuntu mirror at URL instead of the default, which
                        is http://archive.ubuntu.com/ubuntu for official
                        arches and http://ports.ubuntu.com/ubuntu-ports
                        otherwise
    --proxy=URL         Use proxy at URL for cached packages
    --install-mirror=URL
                        Use Ubuntu mirror at URL for the installation only.
                        Apt's sources.list will still use default or URL set
                        by --mirror
    --security-mirror=URL
                        Use Ubuntu security mirror at URL instead of the
                        default, which is http://security.ubuntu.com/ubuntu
                        for official arches and http://ports.ubuntu.com
                        /ubuntu-ports otherwise.
    --install-security-mirror=URL
                        Use the security mirror at URL for installation only.
                        Apt's sources.list will still use default or URL set
                        by --security-mirror
    --components=COMPS  A comma seperated list of distro components to include
                        (e.g. main,universe).
    --ppa=PPA           Add ppa belonging to PPA to the vm's sources.list.
    --lang=LANG         Set the locale to LANG [default: nb_NO.UTF-8]
    --timezone=TZ       Set the timezone to TZ in the vm. [default: UTC]

  Network:
    --domain=DOMAIN     Set DOMAIN as the domain name of the guest [default:
                        defaultdomain].

  Network:
    --ip=ADDRESS        IP address in dotted form [default: dhcp].
    --mac=MAC           MAC address of the guest [default: random].
    --mask=VALUE        IP mask in dotted form [default: based on ip setting].
                        Ignored if ip is not specified.
    --net=ADDRESS       IP net address in dotted form [default: based on ip
                        setting]. Ignored if ip is not specified.
    --bcast=VALUE       IP broadcast in dotted form [default: based on ip
                        setting]. Ignored if ip is not specified.
    --gw=ADDRESS        Gateway (router) address in dotted form [default:
                        based on ip setting (first valid address in the
                        network)]. Ignored if ip is not specified.
    --dns=ADDRESS       DNS address in dotted form [default: based on ip
                        setting (first valid address in the network)] Ignored
                        if ip is not specified.

  libvirt integration:
    --libvirt=URI       Add VM to given URI
    --bridge=BRIDGE     Set up bridged network connected to BRIDGE.
    --network=NETWORK   Set up a network connection to virtual network
                        NETWORK.

  VM settings:
    -m MEM, --mem=MEM   Assign MEM megabytes of memory to the guest vm.
                        [default: 128]
    --cpus=CPUS         Assign NUM cpus to the guest vm. [default: 1]

Caching for faster creation of vm's

Your build machine can cache packages for the process to avoid fetching from the internet thus speeding up the process. This is optional and if you have a decent internet connection it would not matter much.

Apt-cacher seem to work well for this purpose. Start apt-cacher before doing the vmbuilder and point to the cache with the mirror or install-mirror parameter. Apt-cacher uses port 3142.

You can import into cache existing packages to save initial time:

/usr/share/apt-cacher/apt-cacher-import.pl /var/cache/apt/archives/

There are other proxy tools, like apt-proxy, use any if you want this caching.

Customisation of vm (virtual machine)

You will want to have some custom settings on the vm. Set root passwd, add packages etc.

Networking

We use default and get NAT networking. You can interact from your local Ubuntu installation and you virtual machines. See the doc for doing a brigde instead. If you want to route traffic you can use virt-manager with a GUI or do command line with brctl.

Disk and partition layout

The parameters are rootsize, swapsize and optsize. Or you can have a simple text file like this:
root 5000
swap 1000
---
/var 10000

The triple dash separator line makes /var be a separate disk file, /dev/sdb, while root (/) and swap partitions are on /dev/sda.

Add and remove packages

Add and remove packages as needed. Here is what works for me:

--addpkg unattended-upgrades \
--addpkg acpid \
--addpkg ssh \
--addpkg vim \
--addpkg rcs \
--addpkg sendmail \
--addpkg apache2 \
--addpkg libcgi-pm-perl \
--addpkg libapache-session-perl \
--addpkg libapache-htpasswd-perl \
--addpkg wget \
--addpkg subversion \
--addpkg libnet-ldap-perl \
--addpkg libhtml-entities-numbered-perl \
--addpkg libhtml-parser-perl \
--addpkg libhtml-tree-perl \
--addpkg libwww-perl \
--addpkg libauthen-sasl-perl \
--addpkg libi18n-charset-perl \
--addpkg libunicode-maputf8-perl \
--addpkg acpid

Getting Foswiki and setting up a site

We need to create a script file that will be copied into the vm and ran at first boot. The parameter --firstboot scriptfile --firstlogin scriptfile will do this.
#!/bin/bash
# firstlogin.sh 
ip=$(ifconfig eth0 | awk '/inet addr/ { print $2 }' | sed 's/.*://')

echo now visit http://$ip/foswiki/bin/configure

#!/bin/bash
# firstboot.sh
current=Foswiki-1.1.2.tgz
dlurl=http://sourceforge.net/projects/foswiki/files/foswiki/1.1.2
www=/var/www/foswiki

wget $dlurl/$current
# no verbose on untar, takes too long
tar xzf $current
mkdir /var/www/foswiki
cp -a ${current%.tgz}/* /var/www/foswiki/
cd $www
chown -R www-data:www-data $www
htpasswd -bc $www/data/.htpasswd admin secret
chown www-data:www-data  $www/data/.htpasswd

cd $www/bin
cp  LocalLib.cfg.txt LocalLib.cfg
chown www-data:www-data LocalLib.cfg
sed 's|absolute/path/to/your|var/www/foswiki|' -i LocalLib.cfg

sed 's|\(php_admin_flag\)|#\1|' $www/foswiki_httpd_conf.txt > $www/foswiki.conf
sed 's|JohnDoe|admin|' -i $www/foswiki.conf
cp $www/foswiki.conf /etc/apache2/sites-available/foswiki
a2ensite foswiki
/etc/init.d/apache2 reload

If you need to set a proxy for internet connections, it's in /etc/subversion/servers.

We'll stick with the tgz for now.

We can check out current from subversion and do pseudo-install. That would maybe work well, how about mixing linked installed plugins and updating then from configure?

svn co http://svn.foswiki.org/branches/Release01x01@HEAD foswiki
mv foswiki /var/www/foswiki
# etc.. would perhaps be more suitable for dev/test

The checkout can be modified to check out Tags for Foswiki 1.1.3, i.e. the versjon from .tgz file. That would also have known issued and bugs. @HEAD will most likely be stable and have fixes you want. You can also point to trunk for testing bleeding edge.

STATUS: On my machine:
  • Virtual machines are created, from copy/paste text from here:
  • Copy top section (sample) and packages section into a new text file (create.sh), and add the two parameters --firstboot firstboot.sh and --firstlogin firstlogin.sh (need full path to script files, use $(pwd)/firstboot.sh )
    • create the firstboot.sh and firstlogin.sh by copying the two above sections into each file
  • Then you can run the create.sh
  • vm is created
  • start vm, =virsh --connect qemu:///system start vm01
  • firstboot.sh runs at boot
  • login on console, firstlogin.sh runs and gives URL to configure (run via ssh also first time)
  • To find ip of vm without logging in to it first, look in /var/log/syslog. There you find DHCPACK and dnsmasq-dhcp entries.
    • No virsh or other command line to find vm ip?
  • Doing save in configure LocalSite.cfg is created and the wiki is ready to go

TODO:
  • probably something missing, should already hav DBI, LDAP etc...
  • how about that logging....
  • Three files attached to this topic, works here.
  • Make Foswiki 1.1.3 (RC).

A side note on cloning/imaging virtual machines:
  • grub use disk uuid in the configuration, meaning new disk get a different uuid, and that has to be handled (parameter in /etc/default/grub: #GRUB_DISABLE_LINUX_UUID=true, update grub config, /etc/fstab)
  • /etc/udev/rules.d/70-persistent-net.rules ties eth0 to the MAC address when booted for the first time. A clone will therefore create and use eth1 (eth0 binds to the original vm's eth0).

These issues are avoided when using vmbuilder as it is a new install.

Comments

 
I Attachment Action Size Date Who Comment
create.shsh create.sh manage 973 bytes 11 Apr 2011 - 00:53 LarsEik  
firstboot.shsh firstboot.sh manage 855 bytes 11 Apr 2011 - 00:55 LarsEik  
firstlogin.shsh firstlogin.sh manage 131 bytes 11 Apr 2011 - 00:53 LarsEik  
Topic revision: r2 - 11 Apr 2011, LarsEik
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy