This has been updated in Lucid/F13

Intent of this tutorial:

  • use existing tools, i.e. don't roll our own networking stack.
  • change the OS images as little as possible
  • no massive scripts like
  • make everything quick and clear.

How to use lxc containers using libvirt's networking setup but WITHOUT using libvirt/virsh.

If you DO want to use virsh, you may need to check on this: and this will give some hints: and maybe this

Working off

To install (this will install LXC's 6.recent userspace, which is good enough for now)

yum install lxc
    apt-get install lxc    (Lucid or newer, see if you're running Karmic or earlier)

Prerequisites for configuring networking:

apt-get install bridge-utils

Make sure your system supports LXC, everything should appear enabled/green.


Mount the cgroup filesystem. There's a potential downside: now Chrome won't boot.

mkdir -p /cgroup
    echo "none /cgroup cgroup defaults 0 0" >> /etc/fstab
    mount /cgroup

libvirt supports lxc, but virt-manager doesn't yet. Also, libvirt's lxc implementation is pretty hacked up, it's probably not even worth trying yet. (libvirt's LXC is its own implementation, it doesn't use the LXC tools: ) That's OK, we can use libvirt's default NAT network setup and do everything else by hand.

If you don't want to use libvirt's NAT, here is how to manually set it up:

Dead end:

nah...  let's try nat first.
    brctl addbr br0
    brctl setfd br0 0
        -- sets the bridge's forward delay to 0
        -- why?? what is the forward delay? it's 15 by default.
    ifconfig br0 promisc up
    brctl addif br0 eth0
    ifconfig eth0 up
    route add -net default gw br0

Verify Networking:

Let's lean on libvirt to handle all the networking messiness
    First, make sure the devices are available:

    brctl show
        bridge name bridge id       STP enabled interfaces
        pan0        8000.000000000000   no
        virbr0      8000.000000000000   yes

    sudo virsh net-list --all
        Name                 State      Autostart
        default              active     yes

    cat /proc/sys/net/ipv4/ip_forward

    Now -- in theory -- you can use the virtualization network in libvirt like this:
        <interface type='network'>
            <source network='default'/>

If you want to give your guest its own IP address on your lan, you need to use full bridging. That's too complex to go into here.

OK, now let's create the machine. You can:

I'll use debootstrap: TODO: is there a way of doing this without requiring root?

sudo debootstrap –variant=minbase lucid lucid-guest

    You might want to create a tarball so creating a new guest is just a
    matter of unpacking it -- no need to download everything again.
        debootstrap --make-tarball sid.packages.tgz sid
        debootstrap --unpack-tarball sid.packages.tgz sid rootfs

        by default debootstrap sets etc/resolv.conf to the same as on the
        host machine.  On our NATing setup this is correct.
            TODO: not really!  Better would be to use the dnsmasq nameserver
            so when resolv.conf changes we don't have to copy it into every guest.
            All you need is this:
            And yay!!  DHCP sets this automatically so there's no need to
            even worry about resolv.conf.

    what about the etc/hostname file?
        just change it to the new host name.

    TODO: what is with the etc/nologin -> /var/lib/initscripts/nologin broken symlink?
        It's modified on boot up by the initscripts and rmnologin.
        Should it really be a dangling symlink?

    udev doesn't work inside containers so create static devices in /dev.
    Save the /usr/local/bin/lxc-config script from here
    then run it in your bootstrapped directory.
        cd lucid-guest
    TODO: that's a bad name for the script.  It should be lxc-makedev or something like that.

    config file:
    create the lucid-guest.config file.
    "man 5 lxc.conf" for documentation.

        containers always have their own view of the processes in the system,
        their own mount tree, and their own view of interprocess communication
        (IPC) resources.  How do we isolate things beyond that?

    configure the guest:
    chroot lucid-guest
    apt-get install --force-yes -y gpgv
        TODO: why?  what do we need gpgv for?  authenticating dpkg stuff?
    apt-get install -y language-pack-en
    update-locale LANG=”en_US.UTF-8″ LANGUAGE=”en_US.UTF-8″ LC_ALL=”en_US.UTF-8″
    apt-get install openssh-server dhcp3-client
    apt-get install less vim-tiny ping
    apt-get install rsyslog
        # maybe you'd rather have vim-nox instead of vim-tiny
        # lsof strace ...?
    passwd   # set root password
    # you can also use adduser to add users now

    The guest will pull its networking config via DHCP.
    Make /etc/network/interfaces look like this:

        auto lo
        iface lo inet loopback

        auto eth0
        iface eth0 inet dhcp

    I think the virbr0 device automatically has a DHCP server listening
    on it?  That would be super cool if it did.
    It does!  ps ax | grep dnsmasq should show the dhcp server.
    So, the networking set up by debootstrap should just work.

        To change the virbr0 settings:
        virsh net-edit default
        virsh net-destroy default
        virsh net-start default

    lxc-create -f lucid-guest.config -n lucid-guest
    lxc-start lucid-guest
    and it hangs.  crap.

        add --logfile=/dev/stderr --logpriority=DEBUG
        to watch what lxc-start is doing.
        TODO: why can't I watch the console output of the guest?

    lxc-stop -n lucid-guest   (from another terminal)

    rm rootfs.ubuntu/etc/init/tty{4,5,6}.conf
    comment out /proc, /sys and /dev in lib/init/fstab
    still hangs

    ssh priv sep
    touch network ifstate

    change lucid-guest/etc/init/rc-sysinit.conf
        start on filesystem and net-device-up IFACE=lo
    to either:
        start on filesystem # and net-device-up IFACE=lo
        start on startup

    and now it boots.

Network types:



To rsync an lxc container:

    rsync -vrltH --delete -pgo --stats -D --numeric-ids

TODO: how do I assign ram and swap quotas to guests?

    Disk space quotas aren't supported but that's easy enough to fix --
    just use LVM or some sort of BSD quota thing.

TODO: how does the guest reduce the host's volume partway through startup?
        clearly some resource is being shared...   dbus?
    It happens AFTER "init: lxc pre-start process (2) terminated with status 32"
        is printed so I guess it happens as part of guest startup, not while
        preparing the container.

TODO: How do I make it so that shutdown -r and shutdown -h work inside
  the guest?  They both just hang after this:
   * Unmounting local filesystems...                                                                                                                   [ OK ] 
   mount: / is busy
   * Will now restart
  apparently this isn't implemented yet...  more work to be done.

TODO: it's too bad that hitting control-z in the controlling terminal
  doesn't halt all guest processes.  what does it halt?

TODO: the guest's clock is UTC.  this is kinda nice but how do I change it?
    rm -f /etc/localtime
    cp /usr/share/zoneinfo/YOUR_TIME_ZONE /etc/localtime

TODO: DHCP takes a while to get up and running, and initctl list doesn't
  list the interface.  Have I screwed up the config somewhere?
  Those are separate problems.
  It takes somewhere around 3 minutes and 130rx/12tx packets for
     DHCP to configure eth0.  TODO: Why?
    "brctl showstp virbr0" shows forward delay and bridge forward delay of 0.00.
    dhclient isn't even running!
        takes a long ass time to even get around to running ifup eth0!
        what on earth is it waiting for.
    could it be ipv6 related?
    This doesn't seem to be avahi-related (avahi is a usual culprit for slow network bringup).
  TODO: why doesn't initctl show "network-interface (eth0) start/running"

TODO: how do I display console output while the guest is booting?
  The lxc-start command produces the console output after I run
    'shutdown -h' but not before that.  Too bad -- I would really
    like to watch the guest boot!
  is it because of plymouth?
    TODO: how cool would it be if lxc-start would interact with the console
        of the guest being started?

It would be awesome to use btrfs to fire up virtual machines.
    btrfsctl -s ${VPS_DOM}/${dom} ${VPS_TPL}/${tpl}   (dom=domain, tpl=template)

TODO: why am I getting this in my syslog?
    May 11 01:01:45 ubuntu-lucid kernel: [11000.472504] eth0: no IPv rotr rsn
    May 11 01:01:46 ubuntu-lucid kernel: [11000.752503] vethwsaThm: no IPv6 roter rsn
  obviously that's meant to be "eth0: no IPv6 routers present", "vethwsaThm: no IPv6 routers present"
    but why is it garbled?  Should I be worried?
      I don't see this sort of truncation in the host's /var/log/debug file.

OpenVPN tun device: must set it up on the host, not the guest.

TODO: can quota (apt-get install quota,
    support disk quotas?  Seems like it could?

TODO: unnecessary in this image:
        tcpd libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 xauth
    wtf?  how did these get installed?

TODO: we don't need plymouth but for some reason mountall requires it.  huh??
        I could see plymouth requiring mountall but why does mountall require plymouth?
        I guess plymouth has become an integral part of the boot process.
    Later in that thread someone posted a hacked version of mountall with
    no plymouth dependency so you can purge plymouth.  This is BAD.  Don't do it.
    Instead, you can try this:

TODO: is it possible to use lxc without using lxc-create and lxc-destroy?
    Can't I just start a guest by providing a config file, then when that
    guest is stopped, it goes away completely?
        Answer: I think so.  The lxc manpage calls this VOLATILE CONTAINTERS?
            Or maybe that only works for lxc-execute, not lxc-start?

TODO: add lxc to this page?  Would take some investigation.

I've shown you how to connect via console and ssh.

You can use VNC to connect.
    apt-get install vnc4server
    make it start automatically, something like:
        usr/bin/vnc4server :0 -geometry 1024x768 -depth 24
    now, connect via vncviewer <ip>:0

TODO: Can I connect via X?  i.e. run xclock inside the guest, displaying on the host?
    If so, can I use xnest (or something) to open a fully resizable desktop on the guest?

TODO: what are the memory ramifications of lxc-freeze?
    Does freezing a guest allow it to be swapped out completely?

TODO: what happens when you're lxc-monitoring a regex and a new guest
    matching that regex gets started?  Does it automatically get monitored
    too?  (should add this to the lxc-monitor manpage)

TODO: an upstart script to automatically start containers at boot?
    Maybe every config file linked into /var/lxc/at-boot?

TODO: set up a PPA that always has the latest release and nightlies of LXC.

TODO: why not just set lxc.tty=1 ?  Does that affect the number of
    lxc-consoles I can start?  Does the new lxc-enter make this irrelevant?
    also, what are the ramifications of lxc.pseudo?
        You can of course bind a guest's tty to the host.
        - first edit the host's /etc/inittab to turn off getty for that number
        - you probably need an lxc.cgroup.devices.allow for that tty in the guest's config
        - then, on the container, start getty for that number.
        See for more.

TODO: is there some way to set up dnsmasq so I can just use a hostname
    to crawl back up into containers?  Maybe I just need to edit the hosts
    file?  That would be ugly but effective.
  • use deboostrap, febootstrap, or mkarchroot
  • Use Xen's rinse tool
  • Use OpenVZ templates (what does linux-vserver use?)
  • cp -r your current root filesystem. this might be similar:
    • a virtual physical device with its own mac address ? how does it connect to the real world?
    • supports offloading so network performance should be great (unlike veth).
    • can't communicate with other containers running on the same host. (this might change in 2.6.33 with the optional macvlan bridge mode)