I had to spend some time setting up Xen on one of the new Dell servers we bought and while there was some documentation around, I had to constantly refer to the different sources since there wasn’t a complete page which had everything I needed for my particular situation – I have a host machine with Ubuntu Feisty Fawn installed and wanted to install a couple of Xen guest domains on it. One other thing I found lacking in the documentation is the explanation of commands or answers to the question “so why the hell am I doing this?”. I’m not considering LVM at the moment and am happy with loopback disk file images too.

So I think it’s a good idea to write this down somewhere for the next time I have to do the same bloody thing over again on any new servers.

Some caveats before we start:

• Like I said, I’m not gonna talk about LVM for disk images since I know next to nothing about configuring it at the moment. The host machine I have isn’t setup with LVM and I haven’t got the time to look into LVM either.
• I’m quite the networking newbie, so my setup uses the default Xen bridge installed via apt-get. Check out Xen networking on the Xen wiki for more complex network setups.
• I’m not gonna talk about putting a Linux distro other than that of the host machine into the Xen virtual machines. In other words, I have Feisty Fawn for both dom0 and all my domUs.
• I probably did some completely stupid things. Stuff like xen-tools exist to simplify Xen administration, but I haven’t had a chance to use them. Let me know if you spot an error or have a better way of doing something please!

Some terms that deserve explaining (most guides out there just use “DomU” and “Dom0″ without any ceremony, and assume you know what they mean):

• Domain-0 or Dom0: This refers to the host machine OS. You know, the OS of the actual physical server that you have.
• DomU: This refers to a Xen guest domain. A DomU is a single Xen virtual machine. The “U” stands for “unprivileged” I believe.

OK let’s go!

Installing the Xen Server

First off, we need to install the Xen server on the host machine (or Dom0). Thankfully, there is a Xen server package in Feisty Fawn’s apt-get repository.

• Make sure ‘universe’ is enabled in /etc/apt/sources.list and run: apt-get install ubuntu-xen-server
• Reboot the machine. After boot, running sudo xm list should show you Domain-0 (which is the host machine that’s running Xen).
• Edit /etc/xen/xend-config.sxp and uncomment this line:

  (network-script network-bridge)

• Comment out this line:

  (network-script network-dummy)

• Restart xend:

  sudo xend restart

• When you run ‘ifconfig’, you should see the ‘xenbr0′ interface. This is the Xen network bridge.

Creating a base image

Now let’s create a base Xen image. We’ll use this as a template for any future images (domUs).

• Use ‘dd’ to create the images (I place them in /xen-images/):

  dd if=/dev/zero of=/xen-images/feisty_base.img bs=1024k count=4000
  dd if=/dev/zero of=/xen-images/feisty_base_swap.img bs=1024k count=1000

  These commands create image files of 4GB and 1GB for your virtual OS and its swap respectively. ‘count’ is the number of blocks (block size = 1024k).

• Create the filesystem (we’re using ext3 in this case) within the virtual OS image file.

  mkfs.ext3 /xen-images/feisty_base.img

  Answer ‘yes’ to proceed anyway when it complains about the file not being a block device.

• Do the same for the swap file:

  mkswap /xen-images/feisty_base_swap.img

• Now, restrict the permissions on the images so no one else but root can screw around with them.

  chmod 640 /xen-images/feisty_base*

• Create a mount point for the image. We need this to mount the image to install and configure Ubuntu Feisty Fawn the way we like it. We can then replicate this base image for any future virtual machines.

  mkdir /xen-images/mnt

• Mount the base image via the loop device:

  mount -o loop /xen-images/feisty_base.img /xen-images/mnt

  Now we have full access to the base image’s filesystem. What we need to do now is to put a base install of Feisty Fawn on it.

Installing and configuring the base image

Next, we have to put Ubuntu onto the image and configure it to our liking.

• Install debootstrap, a handy tool that bootstraps a basic Debian (Ubuntu in this case) system.

  apt-get install debootstrap

• Bootstrap the base image with ‘feisty’:

  debootstrap feisty /xen-images/mnt

  After this is done you should see the basic Ubuntu file hierarchy:

  ls /xen-images/mnt

• Copy your sources.list into the base image:

  cp /etc/apt/sources.list /xen-images/mnt/etc/apt/

• Copy your kernel modules as well (replace 2.6.19-4-server with whatever kernel version you have):

  cp -a /lib/modules/2.6.19-4-server/ /xen-images/mnt/lib/modules/

• Install the libc6-xen package, which is a Xen-compatible glibc that allows applications to use Thread-Local Storage (TLS).

  apt-get install libc6-xen

  Taken from the Xen FAQ:

  Some modern distributions ship with a ‘TLS’ version of glibc that is not fully compatible with Xen. To use Xen reliably and with maximum performance you must disable the incompatible glibc.

• Configure networking by editing /xen-images/mnt/etc/network/interfaces:

  auto lo
  iface lo inet loopback
  
  # Uncomment this and fill up with your networks settings.
  #auto eth0
  #iface eth0 inet static
  #  address 192.168.0.201
  #  netmask 255.255.255.0
  #  broadcast 192.168.255.255
  # gateway 192.168.0.1
  #  dns-nameservers 192.168.0.1

  Notice how the network settings are commented out – this is because we are leaving the base image as a template from which other images would be made from.

• Edit the hosts file (/xen-images/mnt/etc/hosts):

  127.0.0.1       localhost localhost.localdomain
  127.0.0.1       guest
  
  # The following lines are desirable for IPv6 capable hosts
  ::1     ip6-localhost ip6-loopback
  fe00::0 ip6-localnet
  ff00::0 ip6-mcastprefix
  ff02::1 ip6-allnodes
  ff02::2 ip6-allrouters
  ff02::3 ip6-allhosts

• Give the base image a generic hostname by editing /xen-images/mnt/etc/hostname. In this case I’m just naming it ‘guest’ – actual virtual machines would have sensible hostnames.
• Edit the filesystem table (/xen-images/mnt/etc/fstab).

  proc            /proc           proc    defaults        0       0
  /dev/hda1       /               ext3    defaults,errors=remount-ro 0       1
  /dev/hda2       none            swap    sw              0       0

  Don’t worry, the /dev/hda* would be configured to point to the images you created, not to your actual physical harddisks.

• You’re done with the initial configuration and have enough to bootup the image. Let’s unmount the image:

  umount /xen-images/mnt

• Now, we need to create a Xen config for the base image so we can boot into it. Create a file named /etc/xen/vm_feisty_base.config.example.sxp (you can name it whatever you like of course):

  name = "feisty_base"
  kernel = "/boot/vmlinuz-2.6.19-4-server"
  ramdisk = "/boot/initrd.img-2.6.19-4-server"
  root = "/dev/hda1 ro"
  memory = 512
  disk = ['file:/xen-images/feisty_base.img,hda1,w','file:/xen-images/feisty_base_swap.img,hda2,w']
  
  # Network.
  hostname = "feisty_base"
  vif = ['bridge=xenbr0']
  dhcp = "off"
  ip = "192.168.0.201"
  netmask = "255.255.255.0"
  gateway = "192.168.0.1"

An explanation of the configuration options:

• name – a descriptive name of your image.
• kernel – path to the kernel.
• ramdisk – path to the initrd (initial RAM disk) image.
• root – point this to your host machine’s harddisk where your image files are.
• memory – the amount of memory assigned to the virtual machine in megabytes
• disk – specify a list of block devices that will be export to the virtual machine. In this example, we’re mapping the /xen-images/feisty_base.img image file to /dev/hda1 in the virtual image (same for the swap image). This should match whatever config you did in /xen-images/mnt/etc/fstab.
• hostname – hostname of your virtual machine.
• vif – virtual network interface that your virtual machine will use. In this case, we’re using the default ‘xenbr0′ bridge that Xen installs. For alternative configurations, check out the networking page on the Xen wiki.
• ip – specify your virtual machine’s IP here.
• netmask and gateway – netmask and gateway that your virtual machine should use.

• Now, start the base virtual machine:

  xm create /etc/xen/vm_feisty_base.config.example.sxp -c

  You should see a console (the ‘-c’ option connects you to the console immediately after booting up the virtual machine. Login as root (empty password) – you’ll want to change the password (with ‘passwd’).

• Enable shadowed passwords.

  shadowconfig on

• Create a backup of /etc/network/interfaces and edit /etc/network/interfaces to put in correct values so you can access the Internet from the virtual machine:

  cp /etc/network/interfaces /etc/network/interfaces.bak
  vim /etc/network/interfaces

• Start networking:

  ifup -a

• Install the Ubuntu standard system after running an update:

  apt-get update
  apt-get install ubuntu-standard

• I also like to upgrade any packages that have been updated, if any:

  apt-get upgrade

• Install the OpenSSH server so that you can access the virtual machine via ssh:

  apt-get install ssh

• At this point, do any configuration or installation of packages that you want this base image to have. Remember that we’re going to replicate this base image for future Xen virtual machines so don’t go overboard and install stuff you’re not gonna use.
• Stop networking and put back the original /etc/network/interfaces:

  ifdown -a
  mv /etc/network/interfaces.bak /etc/network/interfaces

• Exit the guest console with Ctrl-]. This will bring you back to dom0′s shell.
• Shutdown the base Xen domU (-w means “wait for the domU to shutdown completely”):

  xm shutdown feisty_base -w

You’re done setting up the base image! Now you have a nicely configured base image.

Creating your first virtual server

The fruits of thy labor are almost at hand – with the base image, we can now make a copy of it for our first Xen virtual machine.

• Make copies of the base image and the swap image (let’s call our virtual server “yuki”):

  cp /xen-images/feisty_base.img /xen-images/yuki.img
  cp /xen-images/feisty_base_swap.img /xen-images/yuki_swap.img

  You may want to resize the base image depending on your disk space needs. (And, of course, you can create your own swap image like we did earlier instead of copying the base image’s swap file.)

• Make a copy of the base Xen configuration file that we created for our base image earlier (you can use whatever name you like):

  cp /etc/xen/vm_feisty_base.config.example.sxp /etc/xen/vm_yuki.config.sxp

• Edit the /etc/xen/vm_yuki.config.sxp file, changing the values to suit your virtual server (you probably want to change at least the ‘name’, ‘disk’, ‘ip, and ‘hostname’ values).

  name = "yuki"
  disk = ['file:/xen-images/yuki.img,hda1,w','file:/xen-images/yuki_swap.img,hda2,w']
  hostname = "yuki"
  ip = "192.168.0.202"

• Now start up your cloned virtual server and log into it like we did earlier with the base image (if you haven’t changed the password, you can login as root with an empty password).

  xm create -c /etc/xen/vm_yuki.config.sxp

• The first thing you should do is to secure your root password (if you haven’t done so in the base image earlier).

  passwd

• Now, you’ll want to change these files:
  • /etc/network/interfaces
  • /etc/hostname
  • /etc/hosts
• With the proper network settings, you should be able to bring up networking to connect to the internet (assuming that your dom0 is connected to the internet).

  ifup -a
  ping google.com

• At this point, you are free to customize your virtual machine as you see fit – install nginx for use as a static file server, Rails, MySQL server, whatever.
• But first, let’s make sure this Xen virtual server starts up automatically on boot. Exit the virtual machine console with Ctrl-]

  ln -s /etc/xen/vm_yuki.config.sxp /etc/xen/auto/

  Any Xen domain configuration placed in /etc/xen/auto/ will be started up automatically on boot (and shutdown when the host machine is shutdown too).

That’s it! You have a functional Xen guest domain that starts up on boot and that can connect to the Internet. Oh before I forget, some useful commands when dealing with the Xen images:

• xm create /path/to/config_file – Starts up the guest domain with the given Xen domU configuration file. (xm create -c to login to the console immediately after booting the image.)
• xm console yuki – Open the console for the ‘yuki’ Xen domain.
• xm shutdown yuki – Shutdown the ‘yuki’ domU.
• xm list – Lists all the Xen domains you have running.

Resizing your virtual machine disk

Let’s say you wanna add 1GB of disk space to your virtual machine (named ‘yuki’).

Note that LVM is usually recommended for Xen setups, so you may wanna take a look at that instead.

• Create a temporary 1GB file:

  dd if=/dev/zero of=/tmp/temp_expand bs=1024k count=1000

• Bring down the virtual machine (I’ve found that a shutdown is needed so that the virtual machine recognizes the increased disk space):

  xm shutdown yuki

• Append the file to the disk image of the virtual machine (/xen-images/yuki.img in this example). Be sure to use ‘>>’ (which means append) – using a single ‘>’ would overwrite the disk image with your temporary file, which you most definitely don’t want!

  cat /tmp/temp_expand >> /xen-images/yuki.img

• Resize the file system:

  resize2fs -f /xen-images/yuki.img

• Startup the virtual machine again:

  xm create -c /etc/xen/vm_yuki.config.sxp

• Remember to login to your virtual machine to verify that the disk space has increased (df -h)