These are some notes that I took while setting up vservers on my network. Please feel free to edit this page and/or copy any useful content to the Linux-VServer wiki.

To set up your machine to create vservers

Read Ubuntu-VServer for how to set up your repositories.

Install the Kernel

First, you need to select and install a vserver-enabled kernel image.

  $ sudo apt-get install util-vserver vserver-debiantools
  $ cat /proc/version
  $ apt-cache search linux-vserver-image
  $ sudo apt-get install linux-vserver-image-NAME

where NAME most closely matches the kernel version printed by /proc/version.

And then reboot your computer and select your vserver-enabled kernel.

Set up /etc

/etc/vservers/newvserver-vars (man newvserver(1) for options):

 DIST='dapper'
 MIRROR="http://us.archive.ubuntu.com/ubuntu"

Also run ln -sf /vservers /etc/vservers/.defaults/vdirbase to specify where your vservers should reside. I keep mine in /vserver but you can place this directory anywhere or leave it in the default place, /var/lib/vservers.

Create a new guest

We'll create a new Ubuntu Dapper image as specified in the setup above. If you want to create a Gentoo guest, follow these instructions.

  newvserver -v --hostname NAME --domain EXAMPLE.COM --interface ETHX --ip IPADDRESS

Select shadow passwords. Also see InitStyles.

There is no need to alias your ethernet device (eth0:1, eth0:2, etc) for the virtual servers. linux-vserver will use the specified device for the specified IP addess no matter what the external settings are. So, just specify the interface and the IP address you want and, as long as that IP address is not already being used, linux-vserver will do the right thing.

Note that guests by default come with a tmpfs limited to 16 MB! This is too small for most real-world use cases so make sure to bump that up to 256MB or so by editing /etc/vservers/NAME/fstab.

Now modify the installed software to be suitable for use as a guest. (WARNING: Using dpkg in this manner will break some dependencies. aptitude is now the Debian-recommended tool to manage packages, so you might consider NOT to purge it, but use it instead to keep your dependency base in order. See http://www.debian.org/doc/FAQ/ch-pkgtools.en.html#s-aptitude):

dpkg --purge ubuntu-minimal alsa-base alsa-utils linux-sound-base libasound2 aptitude \
             dhcp3-client dhcp3-common dosfstools eject ethtool hfsplus hfsutils jfsutils \
             libhfsp0 libiw28 wireless-tools memtest86+ mii-diag ntpdate pcmciautils \
             reiser4progs reiserfsprogs tcpd usbutils wpasupplicant xfsprogs pciutils \
             iproute netbase ifupdown
for link in klogd hwclock.sh urandom umountfs halt reboot mountvirtfs mountall.sh mountnfs.sh;
     do update-rc.d -f $link remove ; update-rc.d -f $link stop 90 2 3 4 5 . ; done

This patch silences the 'cat: /proc/cmdline: No such file or directory' error when shutting down a guest:

--- /etc/init.d/sendsigs        2006-11-12 11:03:35.000000000 -0800
+++ /vservers/scratch1/etc/init.d/sendsigs      2006-11-12 11:03:54.000000000 -0800
@@ -25,6 +25,8 @@
 }
 
 splash_back() {
+    [ -e /proc/cmdline ] || return 0
+
     # Restore usplash
     for x in $(cat /proc/cmdline); do
        case $x in

And this patch silences 'mount: permission denied' when shutting down a guest:

--- umountroot  2006-11-12 11:08:41.000000000 -0800
+++ /vservers/scratch1/etc/init.d/umountroot    2006-11-12 11:15:55.000000000 -0800
@@ -21,7 +21,7 @@
 
 do_stop () {
     [ "$VERBOSE" != no ] && log_action_begin_msg "Mounting root filesystem read-only"
-    mount -n -o remount,ro /
+    mount -n -o remount,ro / >/dev/null 2>&1
     [ "$VERBOSE" != no ] && log_action_end_msg $?
 }

On my box vservers aren't built to use shadow passwords. I don't know why not but it's easy to fix:

 vserver NAME exec pwconv

Download a guest

vserver images can be tarred up just like any other system. So, if you want to try out a different distribution (say, Fedora), you can go to all the trouble to create the guest as above. Or, you can just copy one from someone who's already created one.

http://linux-vserver.org/Downloads#Guest_images

Simply uncompress the archive into your vservers directory and start it up. It should be that easy.

Copy a guest

Building a guest from scratch is a time consuming and network-intensive process, and downloading one is a little scary (is the guy who created it trustworthy?) I find that when I want a new guest, I just copy an existing guest that's close to what I want and tweak it to fit.

Note that the vserver-copy command does not work anymore (this is true as of October 2006), but it's easy enough just to do everything by hand.

  cp -ar /etc/vservers/OLD /etc/vservers/NEW
  cp -ar /vservers/OLD /vservers/NEW
  cd /etc/vservers/NEW
  rm vdir cache run
  ln -s /vservers/NEW vdir
  ln -s /var/run/vservers/NEW run
  vi name interfaces/0/dev interfaces/0/ip uts/nodename context
  cd /vservers/NEW/etc
  vi resolv.conf hostname motd hosts 

Only modify context if your vserver installation doesn't use dynamic context IDs (and if it doesn't, it should).

You can search for all instances of your node name using this command:

  cd /vservers/NEW
  grep -s -D skip -r OLD bin boot etc home initrd lib/ media/ opt/ proc/ root/ sbin/ srv/ sys/ tmp/ usr/ var/

On my Dapper machine, these are the files I must reconfigure after copying the virtual server. I'm using @BLANKVS@ to denote places where I need to substitute the actual machine name.

etc/resolv.conf:search @BLANKVS@
etc/hostname:@BLANKVS@
etc/motd:Debian GNU/Linux (dapper/i686) @BLANKVS@
etc/hosts:192.168.11.5  localhost @BLANKVS@.@BLANKVS@.com @BLANKVS@

And that's it. Your new guest should now be complete and unique.

Delete a vserver

  vserver NAME stop
  rm -rf /etc/vservers/NAME /etc/vservers/.defaults/vdirbase/NAME

Starting/stopping vservers

  vserver NAME start
  vserver NAME stop

Starting/stopping vservers automatically

 echo "default" > /etc/vservers/NAME/apps/init/mark
 /etc/init.d/vserver-default start
 /etc/init.d/vserver-default stop

List running vservers

  vserver-stat
  vps   # (like ps)
  vpstree
  vtop
  openvcp vsmon

Enter a guest

  vserver NAME enter

Run a command in a guest

  vserver NAME exec CMD [ARGS...]
  vserver NAME suexec USER CMD [ARGS...]

Sharing Directories Among Guests

Often you'll want to allow two separate vservers to see the same set of directories. You can do this with bind mounts.

Add a line like this to the vserver's fstab (/etc/vservers/$NAME/fstab):

/path/in/host /path/in/vserver none bind 0 0

path/in/host is relative to the host's root directory, path/in/vserver is relative to the vserver's root directory (of course).

For instance, if vs1 is running a web server, and you want to process its log files in vs2, you'll want to mount vs1's /var/log/httpd directory in vs2 somewhere (say, /var/logfiles):

  • No change is needed for vs1, since we'll just mount vs1's /var/log directory in vs2.
  • /etc/vservers/vs2/fstab should contain:
  /vservers/vs1/var/log /var/logfiles none bind 0 0

Setting up a Private Network

This sets up a network that only exists on the host machine. Guests and the host can use it to talk amongst themselves but packets will never reach the outside world (unless you set up other routing rules).

First, create a dummy network interface on the host. On Debian/Ubuntu you would edit /etc/network/interfaces to add:

   auto dummy0
   iface dummy0 inet static
   address 192.168.9.1
   netmask 255.255.255.0

Then run this command to bring up the interface (the "auto dummy0" line ensures it will be brought up every time the machine boots):

   $ ifup dummy0

Now, add the dummy network to each guest. TODO: show example commands. oldwiki

Helpful Utilities

chbind runs a command, ensuring that it and its children can only see a specific IP address and interface.

  chbind --ip IPADDR/INTERFACE --bcast BROADCAST_ADDR
  chbind --ip eth0:2 /etc/init.d/apache start

reducecap -- reduces the things the current process and its child processes can do (like runn mknod, rawio, etc). Once the security ceiling has been lowered, it can't be raised again.

Link to To Do List

todo


Miscellaneous Musings: What happens when you try to put the vserver to work.

The following added by --Loyeyoung 21:01, 13 April 2007 (PDT) :

Here are some notes from my experience in following the instructions above. . . .

Udev won't install because it can't mknod.

When I set up my guest vserver, a package or three didn't download from the repository. It wasn't a deal-killer, because the script happily continued on its way to set up the guest. I didn't find out about the problem until later when I entered my shiny new vserver and started to work.

root@abraham:~# vserver isaac enter
root@isaac:/#

I commenced to setting up my webserver, etc. (I'll spare you the details.) It was then I saw that some packages weren't working. "No problem," said I, as I fired up aptitude and looked for the broken packages. I adjusted my repository list, updated, and installed the missing packages. But udev didn't get configured, so I couldn't get everything to install without leaving broken dependencies. So I ran "dpkg --configure -a" to see what the matter was. The relevant error message was "mknod: `/lib/udev/devices/ppp': Operation not permitted". After a bit of trial-and-error and a bunch of Google, here's how I solved the problem:

First, I bravely ran back to the host and shut down the vserver:

root@isaac:/# exit
root@abraham:~# vserver isaac stop

Next, I need to configure the vserver to let me use "mknod" and kick it off again:

root@abraham:~# echo CAP_MKNOD > /etc/vservers/isaac/bcapabilities
root@abraham:~# vserver isaac start
root@abraham:~# vserver-stat
CTX   PROC    VSZ    RSS  userTIME   sysTIME    UPTIME NAME
0      115   1.2G 477.1M  18m21s52   2m34s58   8h41m57 root server
49156    1   1.6M   616K   0m00s00   0m00s00   0m16s49 isaac
root@abraham:~#

Swwweeet. Now, go back into the vserver and let dpkg do its work:

root@abraham:~# vserver isaac enter
root@isaac:/# dpkg --configure -a
Setting up udev (093-0ubuntu18.0edgy2) ...
Setting up pcmciautils (014-1ubuntu2) ...
Setting up ubuntu-minimal (1.30) ...
root@isaac:/# 

Now that the crisis was over, I picked back up where I left off getting the vserver all configured. I made two decisions for which the jury is still out. (1) I decided not to purge anything upon which ubuntu-minimal depends. Had this been a pure Debian system, I would have removed many of the same packages that my esteem colleague recommends above. However, my overall laziness and desire to think as little as possible led me to leave them in. I'm very uneasy with breaking up the TCP/IP/Ethernet stack. I figure that the engineers over at Canonical package (X)(K)(E)(U)buntu assuming that the ubuntu-minimal metapackage is always installed, so upgrades will go smoother. Finally, I know of no harm that the alsa, wireless, or the pcmcia packages will do. (2) I did, however, run the lingo beginning with "for link . . . [yada yada yada]", mostly because I don't know exactly what it all does, I figured that my colleague had some reason for doing it, and he probably knows more than I do.

So far, neither decision seems to have caused a problem. I'll come back and let you know if it does.

LAMP installation over VServer

I installed the Ubuntu LAMP server in the usual manner, and learned a few lessons along the way. Here are the issues I ran into . . .

1. Apache won't start on the vserver. / Browsing to vserver's webpages showed the host's webpages instead.

I installed Ubuntu Server without a hitch, except that Apache wouldn't start on the vserver and gave me the following message:

root@isaac:/# apache2ctl start
(98): make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

The problem is that apache running on the host has already bound to all addresses, and the vserver's apache wants to do the same thing. Because they both share the same hardware, the host has locked out the vserver. The solution is to make both servers listen only to their respective IP addresses. For the following, assume that the IP address of the host is 10.0.1.1 and the IP address of the vserver is 10.0.1.2.

root@isaac:/# exit
root@abraham:/etc# echo Listen 10.0.1.1:80 > /etc/apache2/ports.conf
root@abraham:/etc# apache2ctl restart
root@abraham:/etc# vserver isaac enter
root@isaac:/# echo Listen 10.0.1.2: 80 > /etc/apache2/ports.conf
root@isaac:/# apache2ctl restart

NOTE: You may need to fix some other configurations on your computer as a result. Any reference to http://localhost/ will need to be changed to http://your-server.example.com/. Because apache will no longer be listening on localhost, any other HTTP client that's looking for localhost (i.e., URL of http:/localhost/) won't find it. A notable example is if you have a local repository (apt-proxy, apt-mirror, etc.) on the machine and your /etc/apt/source.list file points to the localhost.

2. MySQL server set up.

This turned out to be a way to make the set up more scalable. In the previous section, we ran into the problem of the host and the vserver grabbing all the sockets from each other. This time, I let the host do all the database work. On the vserver, I will point all the database clients to the host so that if the vserver grows up and needs its own box, the original host would still be serving the database. Eventually, I'll put the database server on its own machine with a very fast processor, lots of memory, and little else. So, although I had already installed the MySQL server on the vserver as part of the standard LAMP installation, I purged it.

root@isaac:/# aptitude purge mysql-server mysql-server-5.0

Because aptitude is ever-vigilant about dependencies, several other packages were removed, too. Mailx and postfix were removed, which concerned me, but I figured I would come back to the email a little later when I was ready to set up the email server.

Happy Trails,

Loye Young
http://www.IYCC.net
Laredo, Texas