How to Build a Minimal ZFS NAS without Synology, QNAP, TrueNAS

If you need a basic NAS and don't care about GUI features, it is suprisingly simple to set up a ZFS dataset and share it over the network using Samba.

Scope:

I am using this article to document it for future myself, feel free to adopt it for your needs. Problem with TrueNAS is that it is a full-featured, supposedly enterprise-grade, software suite. While it may be simple to set it up (I've never tried), I just don't need any of the bells and whistles it offers. It's the mismatch between what I need and what it offers; not something inherently wrong with TrueNAS. There is also something to be said about a system you know everything about and not having to rely on yet another thing.

ZFS's best feature that's never explained or written anywhere

ZFS filesystem is self contained. If your OS is nuked suddently, simply take all disks to another machine or install a new OS, install zfs, run zfs import and get back your data. This freedom is underrated and not well understood. It is also not explained anywhere.

It's worth emphasizing: All configuration/details about ZFS is stored on the disks themselves. If you've setup a RAIDZ2 (Raid 6) with 6 disks, they are self contained. Move them to a new machine with zfs tools installed, and simply run zfs import. Boom, they'll show up as RAIDZ2. This is an amazing feature that no matter what happens to the host OS, machine, etc; as long as the disks are not damaged, your data is fine.

Step 1. Locate and Organize Disks

List all disks on a linux machine using lsblk -d -o TRAN,NAME,TYPE,MODEL,SERIAL,SIZE command.

[root@sys ~]# lsblk -d -o TRAN,NAME,TYPE,MODEL,SERIAL,SIZE
TRAN   NAME     TYPE MODEL                        SERIAL           SIZE
       sda      disk Virtual disk                                   40G
nvme   nvme5n1  disk Samsung SSD 990 PRO 4TB      XXXXXXXXXXXXXXX  3.6T
nvme   nvme2n1  disk Samsung SSD 990 PRO 4TB      XXXXXXXXXXXXXXX  3.6T
nvme   nvme6n1  disk Samsung SSD 990 PRO 4TB      XXXXXXXXXXXXXXX  3.6T
nvme   nvme3n1  disk Samsung SSD 990 PRO 4TB      XXXXXXXXXXXXXXX  3.6T

These are brand new NVMe drives from Samsung so they should be completely unallocated.

The disks are also mapped to an ID, running ls -lh /dev/disk/by-id:

[root@sys ~]# ls -lh /dev/disk/by-id
total 0
lrwxrwxrwx. 1 root root 10 May 10 09:34 dm-name-rhel-root -> ../../dm-0
lrwxrwxrwx. 1 root root 10 May 10 09:34 dm-name-rhel-swap -> ../../dm-1
lrwxrwxrwx. 1 root root 10 May 10 09:34 lvm-pv-uuid-DGBpev-Na0C-tY20-YY6E-tpL3-epNA-8Ts3Y0 -> ../../sda3
lrwxrwxrwx. 1 root root 13 May 10 09:34 nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX -> ../../nvme2n1
lrwxrwxrwx. 1 root root 13 May 10 09:34 nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX -> ../../nvme5n1
lrwxrwxrwx. 1 root root 13 May 10 09:34 nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX -> ../../nvme3n1
lrwxrwxrwx. 1 root root 13 May 10 09:34 nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX -> ../../nvme6n1

Notice that they are symlinked to their disk names in /dev/.

We can create a /etc/zfs/vdev_id.conf that maps an alias to these IDs:

[root@sys ~]# vim /etc/zfs/vdev_id.conf

# Add these lines in the vdev_id.conf file
alias nvme0 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX
alias nvme1 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX
alias nvme2 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX
alias nvme3 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX

Run udevadm trigger to set the alias (or you can reboot the machine). We can verify that the aliases have been mapped by running ls -lh /dev/disk/by-vdev:

lrwxrwxrwx.  1 root root  13 May 10 10:28 nvme0 -> ../../nvme2n1
lrwxrwxrwx.  1 root root  13 May 10 10:28 nvme1 -> ../../nvme5n1
lrwxrwxrwx.  1 root root  13 May 10 10:28 nvme2 -> ../../nvme3n1
lrwxrwxrwx.  1 root root  13 May 10 10:28 nvme3 -> ../../nvme6n1

Alias mapping is completely optional, you can if you'd like use the full ID /dev/disk/by-id/nvme-eui.002538414143c248 of the disk when creating the zpool as we will do in the next section. Using an alias makes it nice. However, please don't use /dev/nvme1, /dev/nvme2, ... as the order is not guaranteed, especially if you mount a new drive to the system. Creating a vdev_id.conf ensures that the serial number of the drive is tied to the alias.

Remember when we discussed there is no configuration needed on the host OS? /etc/zfs/vdev_id/conf is not necessary and only used when creating a zpool for convenience. If your OS gets nuked, and you lose vdev_id.conf, it won't matter at all.

Step 2. Create ZPOOL

For this tutorial, I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure. It's up to you if you'd like additional redundancy, RAIDZ2 (RAID 6) would certainly be more risilient.

First, we need to install zfs on the linux machine. Please refer to the OpenZFS documentation on how to install it. It's usually as straight forward as, in my case, dnf install zfs on RHEL 9.

I recommend setting the ashift=12 option when creating the zpool as this is your last chance to do so. Most disks report 512kB sector size to OS due to backwards compatibility reasons, but large disks such as Samsung 990 Pro has a sector size of 4KB or even 8KB. ashift=12 represents a sector size of 4KB which will substantially improve performance.

[root@sys ~]# ls /dev/disk/by-vdev
nvme0 nvme1 nvme2 nvme3
[root@sys ~]# zpool create -o ashift=12 s16z1 raidz1 nvme0 nvme1 nvme2 nvme3
[root@sys ~]# zpool status s16z1
  pool: s16z1
 state: ONLINE
config:

	NAME        STATE     READ WRITE CKSUM
	s16z1       ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    nvme0   ONLINE       0     0     0
	    nvme1   ONLINE       0     0     0
	    nvme2   ONLINE       0     0     0
	    nvme3   ONLINE       0     0     0

errors: No known data errors
[root@sys ~]#

Perfect! I chose s16z1 as the name as it describes the size and type of raid. Most tutorials will use tank as the name, as it relates to pool. Corny, I reject this.

We're not done yet. zpool is a disk abstraction, and zfs is the file system. When we ran zpool create, it created a zfs file system with it.

List all properties of the zfs file system by running: zfs get all s16z1. To check whether our zpool is properly configured with ashift=12, we can run:

[root@sys ~]# zdb | grep ashift

ashift: 12

Before we share the system, configure compression (optionally) as well as the default mount point.

[root@sys ~]# zfs set mountpoint=/mnt/s16z1 s16z1
[root@sys ~]# zfs set compression=lz4 s16z1

Next, let's create a couple of zfs datasets under s16z1 root dataset. We will share them using Samba in the next section.

[root@sys ~]# zfs create s16z1/docs
[root@sys ~]# zfs create s16z1/backups

docs for documents, backups for time machine backup. You can create as many datasets as you'd like. Try to keep them at the top level. If you're wondering what is the difference between just a regular filesystem folder and a dataset—a zfs dataset is a way more than just a folder. You can manage zillion properties of a dataset, encrypt it, send and replicate a dataset, take snapshots, etc.—essentially, the entire ZFS feature set. Therefore, it is a good idea to create individual datasets for large categories of your files. docs or backups is a good abstraction level for a dataset. If you want to send just docs to a another remote server as a backup, you can do that without sending the whole s16z1 root dataset.

We will discuss sharing s16z1/docs as a general purpose share as well as ceating a proper Time Machine storage (for Apple systems) by sharing s16z1/backups with special properties.

Step 3. Share disk on the network

We'll use Samba for this, and the type of file sharing system is orthogonal to ZFS. ZFS doesn't care as long as it is mounted on the host system.

Install Samba:

[root@sys ~]# apt install samba

Create a UNIX user specifically for samba, we'll call it john:

[root@sys ~]# useradd -m john

and create a UNIX password for john:

[root@sys ~]# passwd john

Next, associate the UNIX user john to Samba by creating a Samba password for john . This password will then be used by hosts connecting to the share as SYS\john.

[root@sys ~]# smbpasswd -a john

john is also added to the Samba user group. You can verify john as a SMB user by running:

[root@sys ~]# pdbedit -L -v john

Unix username:        john
NT username:
Account Flags:        [U          ]
User SID:             S-1-5-21-880039843-1994218806-4034623300-1001
Primary Group SID:    S-1-5-21-880039843-1994218806-4034623300-513
Full Name:            john
Home Directory:       \\SYS\john
HomeDir Drive:
Logon Script:
Profile Path:         \\SYS\john\profile
Domain:               SYS
Account desc:
Workstations:
Munged dial:
Logon time:           0
Logoff time:          Wed, 06 Feb 2036 08:06:39 MST
Kickoff time:         Wed, 06 Feb 2036 08:06:39 MST
Password last set:    Fri, 23 Aug 2024 23:46:08 MST
Password can change:  Fri, 23 Aug 2024 23:46:08 MST
Password must change: never
Last bad password   : 0
Bad password count  : 0
Logon hours         : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

To delete john SMB user, pdbedit -x -u john

Now that the UNIX user and Samba user is setup, configuring Samba service is extremely straight forward.

Edit /etc/samba/smb.conf (delete everything and replace with the following):

[docs]
   path = /mnt/s16z1/docs
   browseable = yes
   read only = no
   guest ok = no
   valid users = john
   create mask = 0755
[backups]
   path = /mnt/s16z1/backups
   read only = no
   guest ok = no
   inherit acls = yes
   spotlight = yes
   fruit:aapl = yes
   fruit:time machine = yes
   vfs objects = catia fruit streams_xattr
   valid users = john

Test it out on macOS by mounting it with cmd+K in Finder app and using the following url format: smb://10.0.0.6/docs and smb://10.0.0.6/backups where 10.0.0.6 is the IP of the server sharing the SMB share.

To test it on Debian systems or similar, install apt install smbclient and run:

[root@sys ~]#  smbclient -U john //10.0.0.6/docs -c 'ls'

Mount the smb://10.0.0.6/backups and it will show up as a Time Machine share on macOS. Once mounted, start Time Machine backups by adding the share to macOS > Settings > General > Time Machine.

That's all for now. I plan to write another article expanding on the encryption and very powerful zfs dataset replication features.

Cover image credit to Oracle ZFS systems. God, it is beautiful: https://docs.oracle.com/cd/E78901_01/html/E78910/gqtmb.html