ZFS filesystems quick configuration guide

The ZFS filesystem is an alternative to XFS. While introduced originally in the Solaris OS, ZFS has been ported to ZFS_on_Linux. See also the OpenZFS developers page.

ZFS documentation

Installation of ZFS

We assume an EL8 OS in this page. Following the RHEL-based-distro guide, enable the zfs-release repo from ZFS_on_Linux:

dnf install https://zfsonlinux.org/epel/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm

(The rpm --eval "%{dist}" command simply prints .el8 or similar for your OS).

Use the DKMS kernel module installation method:

dnf install epel-release
dnf install kernel-devel
dnf install zfs

Then activate the ZFS kernel module:

/sbin/modprobe zfs

The alternative kABI-tracking kmod installation method may break the ZFS_on_Linux software after kernel upgrades.

Ansible management of ZFS

See the page on Ansible configuration of Linux servers and desktops. There are Ansible modules for ZFS management:

There does not seem to be any module for zpool management, however.

List disks in the system

The disks in the system must be identified. The following commands are useful for listing disk block devices:

lsscsi --wwn --size

Trying out ZFS

Aaron_Toponce ‘s page has some initial examples.

Create a simple zpool named tank with 4 unused drives (sde sdf sdg sdh):

zpool create tank sde sdf sdg sdh
zpool status tank
df -Ph /tank

Define the mount point for the dataset by adding this option:

-m <mountpoint>

Destroy the testing zpool:

zpool destroy tank

A mirrored pool where all data are mirrored 4 times:

zpool create tank mirror sde sdf sdg sdh

A RAID 0+1 pool with 2+2 disks:

zpool create tank mirror sde sdf mirror sdg sdh

Configuring ZFS

The sections below describe how we have configured ZFS.

Create RAIDZ disks

To setup a RAIDZ pool <poolname> with RAIDZ-1, we use zpool with the “raidz1” VDEV, for example:

zpool create <poolname> raidz1 sde sdf sdg

To setup a RAIDZ pool with RAIDZ-2, we use the “raidz2” VDEV:

zpool create <poolname> raidz2 sde sdf sdg sdh

Adding disks for an SLOG

Read about the Separate Intent Logging Device (SLOG) in the ZFS Intent Log (ZIL) page. The disks should be as fast as possible, such as NVMe or SSD.

To correlate a namespace to a disk device use the following command:


Use /dev/disk/by-id/* disk names with ZFS in stead of /dev/sd* which could become renamed.

Add SLOG and ZIL disks

This section shows how to configure an L2ARC_cache on 2 disk devices.

Assume that the 2 disks /dev/sdb and /dev/sdc will be used. First partition the disks:

parted /dev/sdb unit s mklabel gpt mkpart primary 2048 4G mkpart primary 4G 120G
parted /dev/sdc unit s mklabel gpt mkpart primary 2048 4G mkpart primary 4G 120G

Note: Perhaps it is necessary to use the parted command line and make individual commands like:

parted /dev/sdb
(parted) unit s
(parted) mklabel gpt
(parted) mkpart primary 2048 4G
(parted) mkpart primary 4G 120G
(parted) print
(parted) quit

Use /dev/disk/by-id/* disk names with ZFS in stead of /dev/sd* which could become renamed.

To add 2 disks, for example /dev/sdb and /dev/sdc, to the SLOG, first identify the device WWN names:

ls -l /dev/disk/by-id/* | egrep 'sdb|sdc' | grep wwn

The disks and their partitions partN may be listed as in this example:

/dev/disk/by-id/wwn-0x600508b1001c5db0139e52b3964d02ee -> ../../sdb
/dev/disk/by-id/wwn-0x600508b1001c5db0139e52b3964d02ee-part1 -> ../../sdb1
/dev/disk/by-id/wwn-0x600508b1001c5db0139e52b3964d02ee-part2 -> ../../sdb2
/dev/disk/by-id/wwn-0x600508b1001c45bf78142b67cda9c82b -> ../../sdc
/dev/disk/by-id/wwn-0x600508b1001c45bf78142b67cda9c82b-part1 -> ../../sdc1
/dev/disk/by-id/wwn-0x600508b1001c45bf78142b67cda9c82b-part2 -> ../../sdc2

When the partitions have been created, add the disk partitions 1 and 2 as a ZFS mirrored log and cache, respectively:

zpool add <pool-name> log mirror /dev/disk/by-id/wwn-<name>-part1 /dev/disk/by-id/wwn-<name>-part1 cache /dev/disk/by-id/wwn-<name>-part2 /dev/disk/by-id/wwn-<name>-part2

where the WWN names found above must be used.

Cache and mirror devices can be removed, if necessary, by the zpool-remove command, for example:

zpool remove <pool-name> <mirror>
zpool remove <pool-name> /dev/disk/by-id/wwn-<name>-part2

where the disks are listed by the zpool-status command.

Add SLOG and ZIL on Optane NVDIMM persistent memory

This section show how to configure an L2ARC_cache using NVDIMM 3D_XPoint known as Intel Optane persistent memory DIMM modules.

Partition the NVDIMM disks:

parted /dev/pmem0 unit s mklabel gpt mkpart primary 2048 4G mkpart primary 4G 120G
parted /dev/pmem1 unit s mklabel gpt mkpart primary 2048 4G mkpart primary 4G 120G

and then add the disk partitions 1 and 2 as ZFS cache and log:

zpool add <pool-name> log mirror /dev/pmem0p1 /dev/pmem1p1 cache /dev/pmem0p2 /dev/pmem1p2

ZFS pool capacity should be under 80%

From the Best_practices page:

  • Keep ZFS pool capacity under 80% for best performance. Due to the copy-on-write nature of ZFS, the filesystem gets heavily fragmented.

  • Email reports of capacity at least monthly.

Use this command to view the ZFS pool capacity:

zpool list
zpool list -H -o name,capacity

This crontab job for Monday mornings might be useful:

# ZFS list capacity
0 6 * * 1 /sbin/zpool list

ZFS Compression

Compression is transparent with ZFS if you enable it, see the Compression_and_Deduplication page. This means that every file you store in your pool can be compressed. From your point of view as an application, the file does not appear to be compressed, but appears to be stored uncompressed.

To enable compression on a dataset, we just need to modify the compression property. The valid values for that property are: “on”, “off”, “lzjb”, “lz4”, “gzip”, “gzip[1-9]”, and “zle”:

zfs set compression=lz4 <pool-name>

Monitor compression:

zfs get compressratio <pool-name>

Create ZFS filesystems

You can create multiple separate filesystems within a ZFS pool, for example:

zfs create -o mountpoint=/u/test1 zfspool1/test1

ZFS filesystems can be unmounted and mounted manually by these commands:

zfs unmount ...
zfs mount ...

See man zfs-mount for usage of these commands.

ZFS Snapshots and clones

ZFS snapshots (see man zfs-snapshot) are similar to snapshots with Linux LVM, see Snapshots_and_clones.

You can list snapshots by two methods:

zfs list -t all
cd <mountpoint>/.zfs ; ls -l

You can access the files in a snapshot by mounting it, for example:

mount -t zfs zfstest/zfstest@finbul1-20230131080810 /mnt

The files will be visible in /mnt. Remember to unmount /mnt afterwards.

General snapshot advice:

  • Snapshot frequently and regularly.

  • Snapshots are cheap, and can keep a plethora of file versions over time.

  • Consider using something like the zfs-auto-snapshot script.

ZFS backups

Backup of ZFS filesystems to a remote storage may be done by Sending_and_receiving_filesystems.

A ZFS snapshot can be sent to a remote system like this example:

zfs send tank/test@tuesday | ssh user@server.example.com "zfs receive pool/test"

There are several tools for performing such backups:


See the zfs-autobackup Getting Started Wiki page.

On the remote source machine, we set the autobackup:offsite1 zfs property to true as follows:

[root@remote ~]# zfs set autobackup:offsite1=true <poolname>
[root@remote ~]# zfs get -t filesystem,volume autobackup:offsite1

Running a pull backup from the remote host:

zfs-autobackup -v --ssh-source <remote> offsite1 <poolname>

Since the path to zfs-autobackup is /usr/local/bin and ZFS commands are in /usr/sbin, you must add these paths when running crontab jobs, for example:

0 4 * * * PATH=$PATH:/usr/sbin:/usr/local/bin; zfs-autobackup args...

It is convenient to list all snapshots created by zfs-autobackup:

zfs list -t all

You can mount a snapshot as shown above.

There is a zfs-autobackup troubleshooting page. We have seen the error:

cannot receive incremental stream: destination has been modified since most recent snapshot

which was resolved by zfs_rollback:

zfs rollback <problem-snapshot-name>

Useful ZFS commands

List ZFS filesystems and their properties:

zfs list
zpool list
zpool status <pool-name>
zpool get all <pool-name>
mount -l -t zfs

See the sub-command manual pages for details (for example man zpool-list).

Display logical I/O statistics for ZFS storage pools with zpool-iostat:

zpool iostat -v

Get and set a mountpoint:

zfs get mountpoint <pool-name>
zfs set mountpoint=/u/zfs <pool-name>

E-mail notifications

Using the ZFS Event Daemon (see ZED or man zed), ZFS can send E-mail messages when zpool-events occur. Check the status of ZED by:

systemctl status zed

The ZED configuration file /etc/zfs/zed.d/zed.rc defines variables such as the Email address of the zpool administrator for receipt of notifications; multiple addresses can be specified if they are delimited by whitespace:


You should change root into a system administrator E-mail address, otherwise the domain root@localhost.localdomain will be used. Perhaps you need to do systemctl restart zed after changing the zed.rc file(?).

Scrub and Resilver disks

With ZFS on Linux, detecting and correcting silent data errors is done through scrubbing the disks, see the Scrub_and_Resilver page.

Scrubbing can be made regularly with crontab, for example monthly:

0 2 1 * * /sbin/zpool scrub <pool-name>

or alternatively on machines using Systemd, scrub timers can be enabled on per-pool basis. See the systemd.timer(5) manual page. Weekly and monthly timer units are provided:

systemctl enable zfs-scrub-weekly@<pool-name>.timer --now
systemctl enable zfs-scrub-monthly@<pool-name>.timer --now

Replacing defective disks

Detecting broken disks is explained in the Scrub_and_Resilver page. See the zpool-status if any disks have failed:

zpool status
zpool status -x

Use the zpool-replace command to replace a failed disk, for example disk sde:

zpool replace <pool-name> sde sde
zpool replace -f <pool-name> sde sde

The -f flag may be required in case of errors such as invalid vdev specification.

Hot spare disks will not be added to the VDEV to replace a failed drive by default. You MUST enable this feature. Set the autoreplace feature to on, for example:

zpool set autoreplace=on <pool-name>

Replacing disks can come with big problems, see How to force ZFS to replace a failed drive in place.

ZFS troubleshooting

There is a useful Troubleshooting page which includes a discussion of ZFS_events. Some useful commands are:

zpool events -v
zpool history

Disk quotas for ZFS

Read the zfs-userspace manual page to display space and quotas of a ZFS dataset. We assume a ZFS filesystem <pool-name> and a specific user’s name <username> in the examples below.

Define a user’s disk quota userquota and number-of-files quota userobjquota:

zfs set userquota@<username>=1TB userobjquota@<username>=1M <pool-name>

Using a quota value of none will remove the quota.

The superuser can view the user disk usage and quotas, see the zfs-userspace manual page:

zfs userspace filesystem|snapshot|path|mountpoint
zfs userspace -p filesystem|snapshot|path|mountpoint
zfs userspace -H -p -o name,quota,used,objquota,objused filesystem|snapshot|path|mountpoint

The -p prints parseable numbers, the -H omits the heading. The -o displays only specific columns, this could be used to calculate quota warnings.

Normal users are not allowed to read quotas with the above commands. The following command allows a normal user to print disk usage and quotas:

/usr/sbin/zfs get userquota@$USER,userused@$USER,userobjquota@$USER,userobjused@$USER <pool-name>

Default quotas

Unfortunately, the OpenZFS has no default user quota option, this is only available in the Oracle_Solaris_ZFS implementation, see the defaultuserquota page:

zfs set defaultuserquota=30gb <pool-name>

So with Linux OpenZFS you must set disk quotas individually for each user as shown above.

Tools for managing ZFS disk quotas

We have written some Tools_for_managing_ZFS_disk_quotas providing, for example, commands similar to the standard Linux commands repquota and quota.

NFS sharing ZFS file systems

The zfsprops manual page explains about the NFS sharenfs option:

  • A file system with a sharenfs property of off is managed with the exportfs command and entries in the /etc/ exports file. Otherwise, the file system is automatically shared and unshared with the zfs share and zfs unshare commands.

Alternatively to the exports file, use the zfs set/get sharenfs command to set or list the sharenfs property like in this example:

zfs set sharenfs='rw=' pool1/fs1
zfs get sharenfs pool1/fs1

ZFS will update its /etc/zfs/exports file automatically. Never edit this file directly!

There are some discussions on NFS with ZFS:

NFS tuning

Make sure that a sufficient number of nfsd threads are started by configuring the /etc/nfs.conf file:


This number might be around the number of CPU cores in the server. A systemctl restart nfs-server is required to update the parameters.

For optimizing the transfer of large files, increase the NFS read and write size in the NFS mount command on NFS clients, see man 5 nfs:


Larger values (powers of 2, such as 131072) may also be tried.

See also Optimizing Your NFS Filesystem.

ZFS quotas over NFS

The quota tools for Linux has absolutely no knowledge about ZFS quotas, nor does rquotad, and hence clients mounting via NFS are also unable to obtain this information. See a hack at https://aaronsplace.co.uk/blog/2019-02-12-zfsonline-nfs-quota.html