ZFS Install example

This is (approximately) how I installed to ZFS on FreeBSD.

My setup is booting off a compact flash card in an IDE adapter, I chose this because ZFS on root seemed considerably more risky and would make upgrades more difficult. You could also install to a USB stick and boot off that.

Install to CF/USB

Boot the install CD and do a minimal install to ad0 (for CF) or da0 (for USB).

Once it is done check that it boots properly with the CD removed and so on.

I am using the AHCI CAM driver which is in 8.0 and can be used by putting ahci_load="YES" in /boot/loader.conf and rebooting. It will take precedence over the older ATA driver and ATA disks will be called ada instead of ad. This is optional, and if you don't wish to use it replace ada with ad below.

Enable ZFS by putting zfs_enable="YES" into /etc/rc.conf and rebooting. You can combine this with the previous step to save reboot time.

Creating the ZFS array

Firstly we partition the disks and put a swap partition at the beginning. There are 3 reasons for this, one is that swap on ZFS is not reliable because it can deadlock, and it is not possible to dump to ZFS, also if you need to replace a disk later on (due to failure) it may not be exactly the same size (I have seen instances where the new disks are smaller which means you are stuffed).

Since we are not booting off this array we can use GPT partitions without any worry about the BIOS not understanding it, this also means we get UUID labels for each partition for free.

For each disk you need to run the following steps

• gpart create -s GPT adaX
• gpart add -s 1M -s 4G -t freebsd-swap adaX
• gpart add -t freebsd-zfs adaX

This will create the 2 partitions so you will have /dev/adaXp1 (swap) and /dev/adaXp2 (ZFS).

The "-s 1M" will start the first partition on a 1Mb boundary, this is to give the partitions good alignment. Some newer disks have 4k sectors (vs 512bytes) and their performance is significantly degraded if access is not aligned.

Next we need to work out the UUID for each partition, unfortunately there is no quick way to do this, I found the best way was to query the GEOM configuration sysctl, eg..

[cain 11:55] ~ >sysctl -b kern.geom.conftxt
0 DISK ada4 1000204886016 512 hd 16 sc 63
1 PART ada4p2 991614917120 512 i 2 o 8589952000 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/d97203ec-418f-11df-bcfc-001517e077fb 991614917120 512 i 0 o 0
1 PART ada4p1 8589934592 512 i 1 o 17408 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
2 MIRROR mirror/swap0 8589934080 512
# Snip

# My boot disk (CF card)
0 DISK ad0 4034838528 512 hd 16 sc 63
1 PART ad0s1 4034806272 512 i 1 o 32256 ty freebsd xs MBR xt 165
2 PART ad0s1a 4034806272 512 i 1 o 0 ty freebsd-ufs xs BSD xt 7

We can see ada4 (the line starting with 0) has 2 partitions ada4p1 and ada4p2 (lines starting with 1) and they each have a label (lines starting with 2).

I previously labelled the swap partitions with glabel hence the GPT UUID labels have been replaced by those labels.

glabel label mirror0 ada4p1
glabel label mirror1 ada6p1
etc..

The ZFS partition is /dev/gptid/d97203ec-418f-11df-bcfc-001517e077fb - find all of these for your disks and then create the array

zpool create tank raidz /dev/gptid/d97203ec-418f-11df-bcfc-001517e077fb ...

"tank" is the name of the ZFS pool it can be anything you want. "raidz" is the type of array (could be raidz2 for example). See the zpool man page for more details.

You should now have /tank, eg

[cain 12:10] ~ >df /tank
Filesystem 1K-blocks  Used      Avail Capacity  Mounted on
tank       1832492928    0 1832492928     0%    /tank

You can now create partitions as you like, I would suggest something like the following. Note that you can create more later or even move them around (although be careful if a program has a file open on one you move..)

zfs create tank/var
zfs create tank/usr
zfs create tank/home -o mountpoint=/home
zfs create tank/ports -o mountpoint=/usr/ports -o compression=gzip
zfs create tank/ports/distfiles -o mountpoint=/usr/ports/distfiles -o compression=off
zfs create tank/src -o mountpoint=/usr/src -o compression=gzip

Moving /var and /usr onto ZFS

/var and /usr are still inside /tank, we still need to copy the data over to them. To do this reboot into single user mode and then run..

/etc/rc.d/hostid start
/etc/rc.d/zfs start

tar -C /var -cf - | tar -C /tank/var -xf -
tar -C /usr -cf - | tar -C /tank/usr -xf -

# Now move the old versions out of the way
mv /var /var.orig
mv /usr/usr.orig
# And move the ZFS versions into place
mkdir /var /usr
zfs set mountpoint=/var tank/var
zfs set mountpoint=/usr tank/usr

Now press ctrl-d and the system should boot as normal.

-- DanielOConnor - 08 Aug 2010