🖊️ About 600 words ⏱️ 4 minutes

Intro

I am bulding high redundancy servers (two) to host jails and serve multiple services. I want them to be resilient, and I am OK to tradeoff some I/O performance to gain redundancy, so I made the choice of using ZFS mirror.

Install FreeBSD

A server is made of 4x NVMe drives :

2x 960Go drives for the OS
2x 15TB drives for the storage/data.

$ geom disk list | grep descr
   descr: Micron_7450_MTFDKBA960TFR
   descr: Micron_7450_MTFDKBA960TFR
   descr: KIOXIA KCD8XPUG15T3
   descr: KIOXIA KCD8XPUG15T3

I am bulding two zpool, one for the OS (built by FreeBSD Installer) and another for my jails data/storage. I have used the Auto (ZFS) Parition Type during the installation and mirror for the Pool Type/Disks.

After the installation, I thought I was all set-up to goes behond and start playing with Bastille, but… I had a doubt when I listed all the drives blocks :

$ lsblk
DEVICE         MAJ:MIN SIZE TYPE                                    LABEL MOUNT
nda0             0:93  894G GPT                                         - -
  nda0p1         0:95  260M efi                              gpt/efiboot0 /boot/efi
  <FREE>         -:-   1.0M -                                           - -
  nda0p2         0:97  2.0G freebsd-swap                        gpt/swap0 -
  nda0p3         0:99  892G freebsd-zfs                   gpt/freebsd-zfs <ZFS>
  <FREE>         -:-   324K -                                           - -
nda1             0:101 894G GPT                                         - -
  nda1p1         0:106 260M efi                              gpt/efiboot1 -
  <FREE>         -:-   1.0M -                                           - -
  nda1p2         0:108 2.0G freebsd-swap                 gpt/freebsd-swap SWAP
  nda1p3         0:110 892G freebsd-zfs                   gpt/freebsd-zfs <ZFS>
  <FREE>         -:-   324K -                                           - -
nda2             0:115  14T -                                           - -
nda3             0:121  14T -                                           - -

I realized that /boot/efi was mounted using nda0p1 but not nda1p1 ? To confirm I went ahead and simulated a disk failure: I physically disconnected nda0 drive and tried to boot the system up; at the end this is supposed to work as I am using a ZFS mirror specifically for having every bit replicated to the other drive in case of failure…

Drama when I rebooted

Reboot and Select propre Boot device
Please press the <DEL> key into BIOS setup menu.
or Press the <CTRL-ALT-DEL> key to reboot system.

Why

It was not too difficult to find that I am not the only one, no I am not.

So, the FreeBSD Installer is not mirroring the UEFI partition, in case of drive failure, the system would not find the UEFI,filesystem and the BOOTX64.efi are not there, full stop!

Howto

We have to copy the existing data from nda0p1 to nda1p1.

$ doas newfs_msdos -F 32 -c 1 /dev/nda1p1
$ doas mount /dev/nda1p1 /mnt
$ doas mkdir -p /mnt/efi/boot
$ doas cp -a /boot/loader.efi /mnt/efi/boot/bootx64.efi
$ doas  mkdir -p /mnt/efi/freebsd
$ doas cp -a /boot/loader.efi /mnt/efi/freebsd/loader.efi
$ doas efibootmgr -a -c -l /mnt/efi/freebsd/loader.efi -L FreeBSD
$ doas efibootmgr -v
$ doas umount /mnt

Last step is to comment the efi line in /etc/fstab, in fact if the drive is missing the system would go to Single User Mode, without it would boot fine.

 $ cat /etc/fstab
# Device		Mountpoint	FStype	Options		Dump	Pass#
#/dev/gpt/efiboot0		/boot/efi	msdosfs	rw		2	2

Conclusion

Better to have this tested in pre-production, not that the data would have been lost, ZFS mirror is doing the job, but theses extra maintenance steps would increase greatly the downtime & the stress levels during a disk failure. Not to mention the joke that the server would never have come back to life after a reboot…