I’ve been happily running Plex in a FreeNAS (FreeBSD) Jail for a while, picking up the periodic updates via “pkg update”. But now there’s a snag. Apparently a breaking change was introduced in a recent FreeNAS update and now binaries in the old jail are out of sync with the underlying operating system. The visible symptom is the following error message trying to run pkg update.

pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:11:amd64
Processing entries: 100%
Unable to update repository FreeBSD  
Error updating repositories!

Thankfully we get a detailed and specific error message. A web search using that error output found this discussion thread on FreeNAS user forums, which points to ixsystems issue tracker #27904. Reading the information, this break is not considered a bug in FreeNAS that will be fixed. Instead, the onus is on the user to manually update the jail (pointing to this page on how to do so) or to create a new jail.

After reading over the instructions on updating a jail, it seems easier to create a new jail and migrate my Plex server over to the new instance. None of the media need to be moved because they were a read-only mapping between directories on NAS volume and jail running Plex. It is simple to recreate the mapping after following FreeNAS manual for creating a new jail.

The only parts that needed moving were Plex server’s internal data, which Plex has documented here. Thankfully there are no tricky database operations involved – it is a straightforward copy operation to transfer from the old jail to the new one.

Once the new jail completed running pkg install plexmediaserver, a quick browse through Plex library seems to show everything is in order. The old Plex jail will be kept around in a non-running state for a while, just in case there’s something left behind in the move that we might want retrieve later.

One of the major reasons to set up a FreeNAS machine with ZFS volume is to ensure that network storage data is always available in a redundant manner to ensure everything will still be OK after a hard drive inevitably fails. But before theory can be put into practice, we had to wait for a drive to actually fail. In that sense, today is our lucky day.

A drive failed to respond to some commands and returned errors in the wee hours of the morning. Although it appears to have since recovered and is functioning, we shouldn’t be comforted by the momentary blip – it is almost certainly a sign of things to come if we continue to use this drive. So we’ll replace it instead of waiting for a catastrophic failure.

The instructions to replace a failing drive is covered in the FreeNAS manual. Following the procedures, the drive was taken offline in the Storage/Volume Status screen.

Then we go to Storage/View Disks screen to retrieve the identifying serial number. This ensures that we remove the correct physical drive from the computer by comparing this serial number against the number on the physical label on the drive.

Since this FreeNAS machine does not have hot swap capability, it then had to be shut down for the actual drive replacement. Once the machine restarts, we go back into Storage/Volume Status and select “Replace”. (The button next to “Offline” we clicked earlier.) If there’s any existing data on the replacement drive, FreeNAS will double-check to make sure it’s OK for the replacement drive to be overwritten.

And after that… we wait for the data from the remaining good drive to be replicated to the newly installed replacement drive.

This procedure will take several hours and this time is technically a window of vulnerability – if the remaining good drive fails during this time we’ll lose data. To guard against this, ZFS allows even deeper redundancy by using more than two hard drives. In the case of this server, the data is not critical enough to warrant such protection and we’ll just cross our fingers the remaining drive does not fail during the recovery process.

One problem I had with the FreeNAS plug-in architecture is the lack of a user-friendly update system. (Either it is hard to find, or that it doesn’t exist, it’s not user-friendly either way.) The version of Plex media server I received using the default plug-in installation process was several versions out of date and it looked like I was pretty stuck. In order to update, Google found manual instructions that were… well, let’s just call them ‘nontrivial’.

When I found out Plex media server is also offered as part of the FreeBSD Ports and Packages Collection (tracked via “FreshPorts“) I hoped that might be a much better way to go. I had to manually create the FreeBSD Jail running on my FreeNAS box, which was harder than the built-in plug-in version but was not horrible. Plus, if this goes well, it would be a one-time cost.

After the FreeBSD jail was set up, I opened a shell window into it and typed pkg install plexmediaserver which thankfully took care of downloading and installing all the binaries. The install process also ended with a few paragraphs of instructions on how to configure Plex to automatically start every time that instance of FreeBSD (in my case, the FreeBSD Jail) boots up. Copy-pasting the commands worked as advertised and everything was fairly painless.

I was then able to enjoy my home Plex media server for a few weeks. But there was one unknown: upgrades. I didn’t know how upgrades would work until I needed to perform one, so it’s just a waiting game for a new version of Plex to be released and see how things propagate. This week, the wait was over!

The first notification was from my home Plex media server. Apparently it checks for updates periodically and notifies me when one is available. After waiting a day or two, the update propagated to FreshPorts. Once that occurred, I opened a shell window into the FreeBSD jail running Plex and issued the following commands:

• service plexmediaserver stop to halt the old running Plex.
• pkg update to download the latest database of available packages.
• pkg upgrade to perform updates of any new packages, including Plex.
• service plexmediaserver start to start the new version of Plex.

I reconnect to the web interface of Plex running on my FreeNAS box, and I see the latest version up and running. Success!

After getting Plex plug-in up and running, I started researching the FreeNAS features for hosting other code. I plan to keep my FreeNAS box up and running at all times, as is typical NAS usage, to ensure files are available whenever I want them.  I wanted to know what else I can run on the box at the same time since it is going to be on and consuming electricity anyway.

From highest and most user-friendly level to the lowest, they are:

1) FreeNAS Plug-in: This is how I got started with Plex media server, as it is one of the few plug-ins on the default list. Some flaws with this system were visible immediately. The version of Plex on the default plug-in library is several versions out of date, and there is no user-friendly update mechanism. The user has to go into the FreeBSD jail and update manually. Similarly, in order to access the media files hosted on the same FreeNAS box, the user has to know about manually mapping storage into FreeBSD jails.

It feels like the FreeNAS plug-in ecosystem never matured as much as the creators had hoped. A sentiment confirmed by this page. It explained the reasoning behind the push in FreeNAS 10 (Corral) to move to a system based on Docker containers. Unfortunately, when FreeNAS 10 was abandoned, that push was also put on hold.

Summary: It looks like FreeNAS plug-ins are a dead-end for a deprecated architecture.

2) FreeBSD Jail: Since a FreeNAS plug-in user basically had to know about running a FreeBSD Jail anyway, they might as well learn to work at this more hands-on level rather than depending on the FreeNAS plug-in architecture to sugar-coat it. There are a lot more steps involved, but for popular things, somebody would have posted the list of steps. For example, here’s how to install Plex in a jail. (UPDATE: I’ve learned Plex media server is part of the standard package system and therefore even easier to install: Create a jail, open shell to the jail, type ‘pkg install plexmediaserver‘, done.)

Upside: FreeBSD jails’ isolation protects FreeNAS from some security exploits. The resources consumed by a jail is managed by the same system that manages the rest of FreeNAS and automatically gains all the benefits thereof. Storage in a jail can be mapped to a FreeNAS storage volume to allow (optionally read-only) access.

Downside: FreeBSD jails offers no protection against the nastier kernel-level security exploits.

Summary: FreeBSD Jail makes sense for running relatively trustworthy code that integrates with the volumes on FreeNAS. Plex media server is a good example.

3) Virtual Machine: New in FreeNAS 11 is a feature to create and run virtual machines alongside FreeNAS using the FreeBSD bhyve hypervisor. As of FreeNAS-11-0-U1, this new feature is quite immature. For example, trying to stop a VM in the FreeNAS UI seems to have no effect. I have to go into the administrative shell and use the “bhyvectl‘ command-line utility to stop the VM. As another example, the virtual UEFI boot sequence doesn’t act as some operating systems expect, which can result in the user getting dumped into the UEFI shell. (Something normal users should never see!)

Google pointed me to this page which will help with most Linux distributions that encounter this problem. Thanks to this tip, I got my Ubuntu test server up and running on a FreeNAS VM.

Upside: Full virtual machine isolation will protect against most security exploits.

Downside: Full virtual machine isolation consumes more system resources. Most significantly, the storage is not shared: Space dedicated to a VM is not available to the rest of FreeNAS. And there is no storage mapping so a VM could only access FreeNAS shares over the network interface as if it is physically a different machine.

Summary: Virtual machines make sense for things that do not interact with the rest of FreeNAS, and a good alternative to setting up another physical machine.

Once I had a few simple network file shares set up in FreeNAS, it was enough to do most of what I want in a home network storage device. For a fraction of the cost of a commercial solution like Drobo. Now we can start looking at the less critical fun stuff.

Part of my home media collection stored on my NAS includes various video files that I’ve been carrying around. Most of them were standard-definition video files I recorded off of broadcast television programs. This was done at a time when most people would record to VHS tape. Only the super nerdy types record to computer video.

So I had the files, but I didn’t have a good way to play them straight off a file share. This is where something like Plex comes in. There’s a server-side component that runs on my FreeNAS box, talking to client-side components for various devices. The web client could cast to a Google Chromecast, and the Amazon Fire TV stick has a Plex client app.

For security isolation, FreeNAS runs plugins inside a “jail”. This is a FreeBSD feature that sounds a lot like a Docker container but isn’t a Docker container. This isolation is good default security, but it does mean the Plex Media Server plugin could not see the rest of the FreeNAS box until the user specifies a way for the code inside the jail to see specifically allowed files outside of a jail. I could even specify the storage visibility to be read-only so there would be no accidental manipulation of my video files.

Once I got past the FreeBSD jail mechanics, it was mostly smooth sailing. The only problem came from the large fraction of my files encoded in Windows Media Video format, an old video format that Plex does not support. If I end up deciding I like the Plex experience, I will have to look into doing a bulk re-encode of these old video files.

The authors of FreeNAS tries to make things easy for the user by providing automation tools (“Wizards”) that take care of the fine administrative details without requiring the user to learn all the underlying nuts and bolts of FreeNAS, or FreeBSD, or Linux kernel, etc.

This is especially true for creating network file shares in FreeNAS. It supports many network file sharing protocols. Including the Apple-specific AFP (Apple File Protocol), the Microsoft Windows-based SMB (Server Message Block), the Unix-based NFS (Network File System), plus three others I don’t even understand.

Each of these have their own setup requirements that a casual user like myself is unlikely to get right on our own. So the manual encourages the use of Wizards. (Bold emphasis mine, but I think it should be in the manual!)

FreeNAS® provides a Wizard for creating shares. The Wizard automatically creates the correct type of dataset and permissions for the type of share, sets the default permissions for the share type, and starts the service needed by the share. It is recommended to use the Wizard to create shares.

Windows has a file sharing wizard as well: In Windows file explorer, I would create the folder I want to share, then right-click on that folder to select “Sharing…” This launches the wizard who will then take care of everything else.

Since I was used to the above workflow, I did the same thing in FreeNAS. Create a directory, then run the wizard to share that directory. Unfortunately this results in network shares that were not accessible. (“Access Denied”)

I eventually debugged the problem to my “create a directory” step. Since I had created as an administrator, the permissions on that directory were not set for use by other users. And the FreeNAS wizard did not (or could not) update the permissions properly.

What I needed to do was to launch the FreeNAS network sharing wizard, and tell it to create the directory as part of the network share creation process. This way the directory would be created by the wizard who will properly set the permissions for file sharing.

I did too much and that became unhelpful.

Trust the Wizard.

We’ve established that FreeNAS can mirror the USB flash drive boot device for redundancy. If one of them should fail, the system can be recovered with the other. I hadn’t set out to verify the recovery procedure, but I stumbled into a practice run anyway. In hindsight it’s good to get a feel of my recovery options now when I’m still playing with the system, rather than later when I’m in a panic to recover my data.

Over the past few days I’ve experimented with the boot volume, setting up the mirroring and using the built-in replacement mechanism to retire my USB sticks with checksum errors. After this was done, I unplugged the checksum-error drives (including the one I originally installed FreeNAS on) and rebooted the system.

It failed to boot and ended up at the GRUB recovery menu. Hmm, that’s not good.

I plugged all the old drives back in… and that didn’t improve the situation. Apparently, in the midst of all the mirroring and replacing, I managed to damage GRUB configuration so FreeNAS no longer boots.

Since I expected all the actual operating system files to be OK, I searched for a way to rebuild just the boot loader portion of the USB sticks. There are many ways to do this. The easiest – looking route I took is to perform an in-place upgrade.

I booted up the FreeNAS installation media, and selected the “Install/Upgrade” option. I pointed it to a USB drive that was free of checksum errors but would not boot. The installer detected an existing installation and offered to generate a new boot environment for the existing FreeNAS installation. Sounds perfect! I hit <OK>

Ten minutes later, the upgrade failed due to an error writing boot configuration files the installer wants to add. A quick Google search found a few suggestions on how to fix it using command-line tools I wasn’t familiar with. I decided to try easier things first.

I restarted the installation process and chose the other upgrade option: An upgrade with a disk reformat rather than just generating a new boot environment. The installer will preserve a copy of the FreeNAS configuration but wipe everything else from scratch.

This approached was more destructive and took more time but it worked. My FreeNAS box booted back up as if nothing happened. Success!

Since I went into this FreeNAS project with the expectation to experiment and learn, I followed the recommendation to use commodity USB flash drives as the operating system boot drive despite my skepticism. The previous blog post discussed checksum errors found on some of the USB flash drives I had on hand, and how FreeNAS is able to mitigate the errors by mirroring the operating system across multiple USB sticks. This greatly increased my confidence entrusting the FreeNAS operating system to these USB thumb drives.

Usually these devices are used for ferrying files from one computer to another. They expect to see some files copied onto the drive, then copied off the drive in order. (sequential read/write operations.) Putting them in service as the operating system drive is an entirely different work pattern, with small pieces of data written at unpredictable places and other data retrieved from equally unpredictable places. (random read/write operations.)

I was aware of this difference and it was part of my skepticism using USB sticks as operating system drive. Now that I’m using it, I can see how it works in reality. Thanks to FreeNAS boot device mirroring, I have the confidence to go into this experiment without risking my data.

Outside of errors or outright failures, the other thing I had expected was degraded performance. If the flash memory controller chip is optimized for sequential operations, it might be bad at random operations. Unfortunately there was no way to tell up front. The software running on the flash drive controller isn’t something manufacturers put on the packaging. And every company has their own algorithm.

Since they are so cheap, the easiest thing to do is to just go and try it. The result is clear: some USB flash memory drive controllers are far far better at this workload than others.

Some of this difference is felt immediately, in the time taken for the mirroring duplication process. During this “resilvering” process roughly 2GB of data is copied to the new flash drive. Out of the two drives that had no checksum errors, one of them was able to complete the process in a little over two hours. The other took 26 hours, more than ten times longer!

Their difference is also visible in the FreeNAS system performance reporting page. Since FreeNAS keeps the two drives in sync, it’s easy to see how they respond to identical workloads. The flash drive identified as (da1) breezes through work, never more than 50% busy. Its mirrored sibling (da2) struggles to complete the same work, frequently spiking up to 100% busy.

One of these is much less happy at their new job than the other.

Between the checksum errors and the disk busy graph, I am now much better enlightened on the varying quality of USB flash drives. I had thought of them as all basically equivalent, so I just bought whichever is the cheapest. My attitude has now changed. From here on out, whenever I need to buy USB flash drives, I’ll look for those made by SanDisk.

FreeNAS encourages the use of USB flash drives as the operating system boot drive. This allows FreeNAS to dedicate all of the motherboard SATA connectors for data storage drives. I didn’t think commodity USB flash drives are trustworthy enough to hold the operating system, but I was willing to experiment and be proven wrong.

The very first night, I got worrying news from the nightly system check:

  pool: freenas-boot
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 3K in 0h10m with 0 errors
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0    11

errors: No known data errors

Looking on the bright side, “No known data errors” is comforting, as is the “repaired […] with 0 errors”. It’s nice FreeNAS was able to repair whatever was wrong with my USB stick. I suspect inexpensive commodity USB flash drives frequently encounter errors that are silently corrected by the operating system. Still, an error is an error and it’ll only be a matter of time before I run into a serious problem.

Fortunately, FreeNAS authors had the foresight to make sure a bad boot device does not become a single point of failure. A second one can be added to the system act as a mirror to the boot device. If either of them fails, the other can take over.

Much to my dismay, the second USB stick I tried also encountered a data checksum error. I didn’t have much luck figuring out how to interpret the checksum error code, but I did learn that it is supposed to be zero. The first stick returned 21, the second 26.

I tried a third USB stick and was relieved to finally see a zero checksum. The output below was generated when I ran ‘zpool status’ while the third stick is in the middle of replacing the second stick.

  pool: freenas-boot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress
config:

        NAME             STATE     READ WRITE CKSUM
        freenas-boot     ONLINE       0     0     0
          mirror-0       ONLINE       0     0    21
            da0p2        ONLINE       0     0    21
            replacing-1  ONLINE       0     0     0
              da1p2      ONLINE       0     0    26
              da2p2      ONLINE       0     0     0  (resilvering)

errors: No known data errors

I also found a fourth USB stick that was checksum error-free, so I had it take the place of the first one.

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h29m with 0 errors
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0    21
            da1p2   ONLINE       0     0     0
            da2p2   ONLINE       0     0     0

errors: No known data errors        

Now both boot drives in the mirror set have zero checksum error, but the mirror volume overall still has checksum error 21 from the first USB stick. I’m still learning if that means anything (bad) and what it would take to reset that to zero.

One aspect that was completely neglected in v1 was any kind of an access door. The panels were designed only enough for them to mesh, so the only way to hold things together is to glue everything shut. Completely impractical! The only way to perform any maintenance would be to shatter some acrylic to break the case open.

Now we have two access panels, one in the front and one on the bottom. I tried to see if I could somehow integrate things so there would only need to be one access panel, but never came up with a design that would work well while simultaneously satisfying all the other design goals I’m trying to accomplish.

Each access panel is held in by 4 x #6-32 screws, the standard desktop PC case screw. They fasten into threads I had tapped into the underlying layer of plastic. I should use heat-set inserts for better durability. Unfortunately I didn’t have #6-32 inserts on hand at the time and I thought I would probably build a V3 follow-up anyway.

The front panel opens to allow access to the front chamber where the motherboard and CPU lives. If I wanted to upgrade the memory or switch out the SATA cables, I could do so through the front panel opening.

The bottom panel opens to allow access to the rear chamber. The two hard drives and the power supply are all installed through the bottom opening. The drives can be individually replaced without fighting through too many pieces of cable. The power supply can also be replaced through the opening exposed by the bottom panel.

Now that FreeNAS Box v2 is up and running, let’s do a size comparison to see how things have changed. The width dimension was a regression: v2 is wider by 3 centimeters. The real space requirement increased even more than that, because the v2 air intakes are on the sides.  So it needs additional room to the left and right of it in order to avoid blocking those intakes. In contrast, v1 with its bottom intake would be OK sitting flush against objects to its left and right.

Looking at the final results in reality, I think I can rearrange a few things to reduce the width by 1 cm. Something to consider for a potential FreeNAS Box v3.

The greatest improvement is in the height which has been reduced by 7 cm thanks to the elimination of the lower air intake cavity. The advantage is somewhat reduced when in use, because the power cable now sticks straight up and adds a bit to the required height. If this becomes a serious problem, though, I could always switch to an power cable with a right-angle plug. This will allow the box to fit in a shelf only 28 cm high, leaving barely enough for proper heat exhaust. Looking over the results, I think I can safely reduce the bottom of the case by 1 cm and everything will still fit, another change for the potential v3.

And finally, the depth is reduced by 3.5 cm. Real world improvement is even more, because now the back of the box can sit flush against the wall in a way that v1 could not.

The reduced height and depth has more than compensated for the increase in width, v2 is overall a more compact space-efficient package than the v1 design.