ZFS filesystems quick configuration guide 

The ZFS filesystem is an alternative to XFS. While introduced originally in the Solaris OS, ZFS has been ported to ZFS_on_Linux. See also the OpenZFS developers page.

ZFS documentation 

Getting_Started with ZFS including an RHEL-based-distro guide.
First time OpenZFS users are encouraged to check out Aaron_Toponce ’s excellent documentation.
Best_practices and caveats.
OpenZFS_Newcomers documentation and FAQ.
ZFS Basic Concepts.
ZFS man-pages.
Lustre and ZFS:
- Lustre uses ZFS, see https://wiki.lustre.org/ZFS.
- A ZFS setup for Lustre.
- A JBOD Setup page.
Zpool Concepts.
ZFS 101—Understanding ZFS storage and performance and ZFS fans, rejoice—RAIDz expansion will be a thing very soon.
ZFS_checksums are a key feature of ZFS and an important differentiator for ZFS over other RAID implementations and filesystems.

Installation of ZFS 

We assume an EL8 OS in this page. Following the RHEL-based-distro guide, enable the zfs-release repo from ZFS_on_Linux:

dnf install https://zfsonlinux.org/epel/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm

(The rpm --eval "%{dist}" command simply prints .el8 or similar for your OS).

Use the DKMS kernel module installation method:

dnf install epel-release
dnf install kernel-devel
dnf install zfs

Then activate the ZFS kernel module:

/sbin/modprobe zfs

The alternative kABI-tracking kmod installation method may break the ZFS_on_Linux software after kernel upgrades.

Ansible management of ZFS 

See the page on Ansible configuration of Linux servers and desktops. There are Ansible modules for ZFS management:

https://docs.ansible.com/ansible/2.9/modules/zfs_module.html
Additional ZFS modules are found at https://docs.ansible.com/ansible/latest/collections/community/general/

There does not seem to be any module for zpool management, however.

List disks in the system 

The disks in the system must be identified. The following commands are useful for listing disk block devices:

lsblk
lsscsi --wwn --size

Trying out ZFS 

Aaron_Toponce ‘s page has some initial examples.

Create a simple zpool named tank with 4 unused drives (sde sdf sdg sdh):

zpool create tank sde sdf sdg sdh
zpool status tank
df -Ph /tank

Define the mount point for the dataset by adding this option:

-m <mountpoint>

Destroy the testing zpool:

zpool destroy tank

A mirrored pool where all data are mirrored 4 times:

zpool create tank mirror sde sdf sdg sdh

A RAID 0+1 pool with 2+2 disks:

zpool create tank mirror sde sdf mirror sdg sdh

Configuring ZFS 

The sections below describe how we have configured ZFS.

Create RAIDZ disks 

To setup a RAIDZ pool <poolname> with RAIDZ-1, we use zpool with the “raidz1” VDEV, for example:

zpool create <poolname> raidz1 sde sdf sdg

To setup a RAIDZ pool with RAIDZ-2, we use the “raidz2” VDEV:

zpool create <poolname> raidz2 sde sdf sdg sdh

Adding disks for an SLOG 

Read about the Separate Intent Logging Device (SLOG) in the ZFS Intent Log (ZIL) page. The disks should be as fast as possible, such as NVMe or SSD.

To correlate a namespace to a disk device use the following command:

lsblk

Use /dev/disk/by-id/* disk names with ZFS in stead of /dev/sd* which could become renamed.

Add SLOG and ZIL disks 

This section shows how to configure an L2ARC_cache on 2 disk devices.

Assume that the 2 disks /dev/sdb and /dev/sdc will be used. First partition the disks:

parted /dev/sdb unit s mklabel gpt mkpart primary 2048 4G mkpart primary 4G 120G
parted /dev/sdc unit s mklabel gpt mkpart primary 2048 4G mkpart primary 4G 120G

Note: Perhaps it is necessary to use the parted command line and make individual commands like:

parted /dev/sdb
(parted) unit s
(parted) mklabel gpt
(parted) mkpart primary 2048 4G
(parted) mkpart primary 4G 120G
(parted) print
(parted) quit

Use /dev/disk/by-id/* disk names with ZFS in stead of /dev/sd* which could become renamed.

To add 2 disks, for example /dev/sdb and /dev/sdc, to the SLOG, first identify the device WWN names:

ls -l /dev/disk/by-id/* | egrep 'sdb|sdc' | grep wwn

The disks and their partitions partN may be listed as in this example:

/dev/disk/by-id/wwn-0x600508b1001c5db0139e52b3964d02ee -> ../../sdb
/dev/disk/by-id/wwn-0x600508b1001c5db0139e52b3964d02ee-part1 -> ../../sdb1
/dev/disk/by-id/wwn-0x600508b1001c5db0139e52b3964d02ee-part2 -> ../../sdb2
/dev/disk/by-id/wwn-0x600508b1001c45bf78142b67cda9c82b -> ../../sdc
/dev/disk/by-id/wwn-0x600508b1001c45bf78142b67cda9c82b-part1 -> ../../sdc1
/dev/disk/by-id/wwn-0x600508b1001c45bf78142b67cda9c82b-part2 -> ../../sdc2

When the partitions have been created, add the disk partitions 1 and 2 as a ZFS mirrored log and cache, respectively:

zpool add <pool-name> log mirror /dev/disk/by-id/wwn-<name>-part1 /dev/disk/by-id/wwn-<name>-part1 cache /dev/disk/by-id/wwn-<name>-part2 /dev/disk/by-id/wwn-<name>-part2

where the WWN names found above must be used.

Cache and mirror devices can be removed, if necessary, by the zpool-remove command, for example:

zpool remove <pool-name> <mirror>
zpool remove <pool-name> /dev/disk/by-id/wwn-<name>-part2

where the disks are listed by the zpool-status command.

Add SLOG and ZIL on Optane NVDIMM persistent memory 

This section show how to configure an L2ARC_cache using NVDIMM 3D_XPoint known as Intel Optane persistent memory DIMM modules.

Partition the NVDIMM disks:

parted /dev/pmem0 unit s mklabel gpt mkpart primary 2048 4G mkpart primary 4G 120G
parted /dev/pmem1 unit s mklabel gpt mkpart primary 2048 4G mkpart primary 4G 120G

and then add the disk partitions 1 and 2 as ZFS cache and log:

zpool add <pool-name> log mirror /dev/pmem0p1 /dev/pmem1p1 cache /dev/pmem0p2 /dev/pmem1p2

ZFS pool capacity should be under 80%

From the Best_practices page:

Keep ZFS pool capacity under 80% for best performance. Due to the copy-on-write nature of ZFS, the filesystem gets heavily fragmented.
Email reports of capacity at least monthly.

Use this command to view the ZFS pool capacity:

zpool list
zpool list -H -o name,capacity

This crontab job for Monday mornings might be useful:

# ZFS list capacity
0 6 * * 1 /sbin/zpool list

Compression is transparent with ZFS if you enable it, see the Compression_and_Deduplication page. This means that every file you store in your pool can be compressed. From your point of view as an application, the file does not appear to be compressed, but appears to be stored uncompressed.

To enable compression on a dataset, we just need to modify the compression property. The valid values for that property are: “on”, “off”, “lzjb”, “lz4”, “gzip”, “gzip[1-9]”, and “zle”:

zfs set compression=lz4 <pool-name>

Monitor compression:

zfs get compressratio <pool-name>

Create ZFS filesystems 

You can create multiple separate filesystems within a ZFS pool, for example:

zfs create -o mountpoint=/u/test1 zfspool1/test1

ZFS filesystems can be unmounted and mounted manually by these commands:

zfs unmount ...
zfs mount ...

See man zfs-mount for usage of these commands.

ZFS Snapshots and clones 

ZFS snapshots (see man zfs-snapshot) are similar to snapshots with Linux LVM, see Snapshots_and_clones.

You can list snapshots by two methods:

zfs list -t all
cd <mountpoint>/.zfs ; ls -l

You can access the files in a snapshot by mounting it, for example:

mount -t zfs zfstest/zfstest@finbul1-20230131080810 /mnt

The files will be visible in /mnt. Remember to unmount /mnt afterwards.

To destroy a snapshot:

zfs destroy [-Rdnprv] filesystem|volume@snap[%snap[,snap[%snap]]]

see man zfs-destroy.

General snapshot advice:

Snapshot frequently and regularly.
Snapshots are cheap, and can keep a plethora of file versions over time.
Consider using something like the zfs-auto-snapshot script.

ZFS backups 

Backup of ZFS filesystems to a remote storage may be done by Sending_and_receiving_filesystems.

A ZFS snapshot can be sent to a remote system like this example:

zfs send tank/test@tuesday | ssh user@server.example.com "zfs receive pool/test"

There are several tools for performing such backups:

zfs-autobackup creates ZFS snapshots on a source machine and then replicates those snapshots to a target machine via SSH.
https://serverfault.com/questions/842531/how-to-perform-incremental-continuous-backups-of-zfs-pool

zfs-autobackup 

See the zfs-autobackup Getting Started Wiki page.

On the remote source machine, we set the autobackup:offsite1 zfs property to true as follows:

[root@remote ~]# zfs set autobackup:offsite1=true <poolname>
[root@remote ~]# zfs get -t filesystem,volume autobackup:offsite1

Running a pull backup from the remote host:

zfs-autobackup -v --ssh-source <remote> offsite1 <poolname>

Since the path to zfs-autobackup is /usr/local/bin and ZFS commands are in /usr/sbin, you must add these paths when running crontab jobs, for example:

0 4 * * * PATH=$PATH:/usr/sbin:/usr/local/bin; zfs-autobackup args...

It is convenient to list all snapshots created by zfs-autobackup:

zfs list -t all

You can mount a snapshot as shown above.

There is a zfs-autobackup troubleshooting page. We have seen the error:

cannot receive incremental stream: destination has been modified since most recent snapshot

which was resolved by zfs_rollback:

zfs rollback <problem-snapshot-name>

Useful ZFS commands 

List ZFS filesystems and their properties:

zfs list
zpool list
zpool status <pool-name>
zpool get all <pool-name>
mount -l -t zfs

See the sub-command manual pages for details (for example man zpool-list).

Display logical I/O statistics for ZFS storage pools with zpool-iostat:

zpool iostat -v

Get and set a mountpoint:

zfs get mountpoint <pool-name>
zfs set mountpoint=/u/zfs <pool-name>

E-mail notifications 

Using the ZFS Event Daemon (see ZED or man zed), ZFS can send E-mail messages when zpool-events occur. Check the status of ZED by:

systemctl status zed

The ZED configuration file /etc/zfs/zed.d/zed.rc defines variables such as the Email address of the zpool administrator for receipt of notifications; multiple addresses can be specified if they are delimited by whitespace:

ZED_EMAIL_ADDR="root"

You should change root into a system administrator E-mail address, otherwise the domain root@localhost.localdomain will be used. Perhaps you need to do systemctl restart zed after changing the zed.rc file(?).

Scrub and Resilver disks 

With ZFS on Linux, detecting and correcting silent data errors is done through scrubbing the disks, see the Scrub_and_Resilver page.

Scrubbing can be made regularly with crontab, for example monthly:

0 2 1 * * /sbin/zpool scrub <pool-name>

or alternatively on machines using Systemd, scrub timers can be enabled on per-pool basis. See the systemd.timer(5) manual page. Weekly and monthly timer units are provided:

systemctl enable zfs-scrub-weekly@<pool-name>.timer --now
systemctl enable zfs-scrub-monthly@<pool-name>.timer --now

Replacing defective disks 

Detecting broken disks is explained in the Scrub_and_Resilver page. See the zpool-status if any disks have failed:

zpool status
zpool status -x

The RHEL page How to rescan the SCSI bus to add or remove a SCSI device without rebooting the computer has useful information about Adding a Storage Device or a Path. You may scan the system for disk changes using /usr/bin/rescan-scsi-bus.sh from the sg3_utils package. Unfortunately, it may sometimes be necessary to reboot the server so that the OS will discover the replaced /dev/sd??? disk device.

Use the zpool-replace command to replace a failed disk, for example disk sde:

zpool replace <pool-name> sde sde
zpool replace -f <pool-name> sde sde

The -f flag may be required in case of errors such as invalid vdev specification.

Hot spare disks will not be added to the VDEV to replace a failed drive by default. You MUST enable this feature. Set the autoreplace feature to on, for example:

zpool set autoreplace=on <pool-name>

Replacing disks can come with big problems, see How to force ZFS to replace a failed drive in place.

ZFS troubleshooting 

There is a useful Troubleshooting page which includes a discussion of ZFS_events. Some useful commands are:

zpool events -v
zpool history

If a normal user, and also the daily logwatch scripts, tries to execute zpool status an error message may appear:

Permission denied the ZFS utilities must be run as root

This seems to be a Systemd issue, see permissions issues with openzfs #28653. There seems to be a fix in Udev vs tmpfiles take 2 #28732, however, this has not been tested on EL8 yet.

Disk quotas for ZFS 

Read the zfs-userspace manual page to display space and quotas of a ZFS dataset. We assume a ZFS filesystem <pool-name> and a specific user’s name <username> in the examples below.

Define a user’s disk quota userquota and number-of-files quota userobjquota:

zfs set userquota@<username>=1TB userobjquota@<username>=1M <pool-name>

Using a quota value of none will remove the quota.

We have written some Tools_for_managing_ZFS_disk_quotas providing, for example, commands similar to the standard Linux commands repquota and quota.

The superuser can view the user disk usage and quotas, see the zfs-userspace manual page:

zfs userspace filesystem|snapshot|path|mountpoint
zfs userspace -p filesystem|snapshot|path|mountpoint
zfs userspace -H -p -o name,quota,used,objquota,objused filesystem|snapshot|path|mountpoint

The -p prints parseable numbers, the -H omits the heading. The -o displays only specific columns, this could be used to calculate quota warnings.

Normal users are not allowed to read quotas with the above commands. The following command allows a normal user to print disk usage and quotas:

/usr/sbin/zfs get userquota@$USER,userused@$USER,userobjquota@$USER,userobjused@$USER <pool-name>

Default quotas 

Unfortunately, the OpenZFS has no default user quota option, this is only available in the Oracle_Solaris_ZFS implementation, see the defaultuserquota page:

zfs set defaultuserquota=30gb <pool-name>

So with Linux OpenZFS you must set disk quotas individually for each user as shown above.