Learning About Airflow From Three Years Of Dust

Three years ago I started this custom computer case project to build something tailored to run FreeNAS. The primary purpose was to get some hands-on time working with laser-cut acrylic, and I learned a lot building it. Now that I’ve decided to upgrade my home server to a different configuration, there will be too much hardware to fit in this box. I’ll start with a commodity PC tower case but I might build another custom case later. Either way, this little acrylic box will be retired.

Since the computer has been sitting in a corner unobtrusively serving up files for my home network, it has also collected three years of dust. The top layer is not particularly interesting, as they were deposited by gravity. The remainder, though, serve as indicator for airflow through the system and serves as a record of comparison against my intended airflow design for the box.

The biggest lesson for me was that convection played a much smaller role than I had expected. Most of the dust indicating flow was proportional to the size of their air channels, there’s no visible sign of convection altering the flow. The most visible example is the ring of dust near the CPU fan on my front panel. I had expected it to be slightly teardrop-shaped to reflect heat rising, but it is almost a perfect circle.

The most unexpected cluster of dust is on the auxiliary CPU power cable, running to the right side of the CPU fan alongside the USB wires. It appears most of the dust there were carried by air drawn in through the front panel gap. I hypothesize that, since it is a very narrow gap, airflow through that route is slowed and thus more likely to deposit dust on that cable bundle.

There were a few minor smudgest of dust whose origins are a mystery. Two up top near the PSU fan, and one on the bottom at the rear end of the PSU. I’m curious what they were, but their fine dust particle size implies they were not a significant factor, so I’m content to leave them as mysteries for now. Maybe they’ll make sense for me in the future once I learn more about designing for airflow. In order to preserve this information (all this dust will be disturbed and cleaned up when I disassemble this box) I shot a video for future reference:

(Cross-posted to Hackaday.io)

FreeNAS Box Decommissioned After Three Years

I’ve decided to retire my current home server running FreeNAS. It has worked well for three trouble-free years and will likely continue working for a few more. But I have enough motivations for an upgrade beyond its current capabilities.

First, I learned that FreeNAS has been making more and more use of its boot drive in its recent releases. At one point all the SATA ports on a FreeNAS box could be dedicated to storage devices, because FreeNAS itself is happy to boot from a USB flash drive, load to RAM, and run from there. Thus the boot drive is touched very little, minimizing wear on flash memory. However, FreeNAS documentation explained this has not been the case for several years. I have yet to run into any problems with the USB flash drives I’ve been using as mirrored boot volumes, but after three years of service I decided not to wait until problems crop up.

When looking at a boot drive for a modern operating system, my default choice is to use a solid state drive. SSDs were still an expensive luxury when I first started playing with FreeNAS, but they are now quite affordable and there’s been enough hardware churn for a few of my own SSDs to drop out of circulation and thus available for use. My first two Intel X25-M SSDs still report over 85% of wear life remaining. Their modest 80GB capacity is pretty cramped for modern desktop use, but quite spacious for a FreeNAS system drive. That capacity also means a lot of elbow room for flash wear-leveling.

The downside, of course, is that I need a SATA port on the motherboard to connect to my old but still functional X-25M. In order to have a X-25M as my FreeNAS system drive, I had to upgrade beyond this MSI AM1I motherboard with only two SATA ports.

Another motivation was an interest in hosting more functionality on the home server. While code with FreeBSD support can run in a jail, I needed virtual machines to run non-FreeBSD code such as ROS. When I started looking at FreeNAS, virtual machine support via bhyve was an experimental new feature. It has since grown to be a mature part of the feature set. With virtualization I can use the same physical box to host other software projects.

But a virtual machine also locks out a portion of system memory as any RAM allocated to the virtual machine is not available to the rest of FreeNAS. I have many 8GB DDR3 memory modules, but there were only two memory slots on an AM1I motherboard for 16GB. Moving to a motherboard with 4 memory slots will double the available memory to 32GB, plenty of room for playing with VMs.

With these points in mind, I powered off my homebuilt FreeNAS box built of laser-cut acrylic. The two storage drives will be moved to a commodity PC tower case. But before I take it all apart, I wanted to make note of a few observations from this computer’s three years of sitting on a shelf quietly running FreeNAS.

Resolving Plex Update Failure in FreeNAS Jail “repository FreeBSD contains packages for wrong OS version”

plex-logo-e1446990678679I’ve been happily running Plex in a FreeNAS (FreeBSD) Jail for a while, picking up the periodic updates via “pkg update”. But now there’s a snag. Apparently a breaking change was introduced in a recent FreeNAS update and now binaries in the old jail are out of sync with the underlying operating system. The visible symptom is the following error message trying to run pkg update.

pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:11:amd64
Processing entries: 100%
Unable to update repository FreeBSD  
Error updating repositories!

Thankfully we get a detailed and specific error message. A web search using that error output found this discussion thread on FreeNAS user forums, which points to ixsystems issue tracker #27904. Reading the information, this break is not considered a bug in FreeNAS that will be fixed. Instead, the onus is on the user to manually update the jail (pointing to this page on how to do so) or to create a new jail.

After reading over the instructions on updating a jail, it seems easier to create a new jail and migrate my Plex server over to the new instance. None of the media need to be moved because they were a read-only mapping between directories on NAS volume and jail running Plex. It is simple to recreate the mapping after following FreeNAS manual for creating a new jail.

The only parts that needed moving were Plex server’s internal data, which Plex has documented here. Thankfully there are no tricky database operations involved – it is a straightforward copy operation to transfer from the old jail to the new one.

Once the new jail completed running pkg install plexmediaserver, a quick browse through Plex library seems to show everything is in order. The old Plex jail will be kept around in a non-running state for a while, just in case there’s something left behind in the move that we might want retrieve later.

FreeNAS Successfully Recovered From Failed Drive

One of the major reasons to set up a FreeNAS machine with ZFS volume is to ensure that network storage data is always available in a redundant manner to ensure everything will still be OK after a hard drive inevitably fails. But before theory can be put into practice, we had to wait for a drive to actually fail. In that sense, today is our lucky day.

FreeNAS Critical Alert Box

A drive failed to respond to some commands and returned errors in the wee hours of the morning. Although it appears to have since recovered and is functioning, we shouldn’t be comforted by the momentary blip – it is almost certainly a sign of things to come if we continue to use this drive. So we’ll replace it instead of waiting for a catastrophic failure.

The instructions to replace a failing drive is covered in the FreeNAS manual. Following the procedures, the drive was taken offline in the Storage/Volume Status screen.

FreeNAS Volume Status

Then we go to Storage/View Disks screen to retrieve the identifying serial number. This ensures that we remove the correct physical drive from the computer by comparing this serial number against the number on the physical label on the drive.

FreeNAS Storage View Disks

Since this FreeNAS machine does not have hot swap capability, it then had to be shut down for the actual drive replacement. Once the machine restarts, we go back into Storage/Volume Status and select “Replace”. (The button next to “Offline” we clicked earlier.) If there’s any existing data on the replacement drive, FreeNAS will double-check to make sure it’s OK for the replacement drive to be overwritten.

FreeNAS Force Replace

And after that… we wait for the data from the remaining good drive to be replicated to the newly installed replacement drive.

FreeNAS Resilver In Progress

This procedure will take several hours and this time is technically a window of vulnerability – if the remaining good drive fails during this time we’ll lose data. To guard against this, ZFS allows even deeper redundancy by using more than two hard drives. In the case of this server, the data is not critical enough to warrant such protection and we’ll just cross our fingers the remaining drive does not fail during the recovery process.

Installing and Updating Plex Media Server in a FreeBSD Jail (FreeNAS) via ‘pkg’

plex-logo-e1446990678679One problem I had with the FreeNAS plug-in architecture is the lack of a user-friendly update system. (Either it is hard to find, or that it doesn’t exist, it’s not user-friendly either way.) The version of Plex media server I received using the default plug-in installation process was several versions out of date and it looked like I was pretty stuck. In order to update, Google found manual instructions that were… well, let’s just call them ‘nontrivial’.

When I found out Plex media server is also offered as part of the FreeBSD Ports and Packages Collection (tracked via “FreshPorts“) I hoped that might be a much better way to go. I had to manually create the FreeBSD Jail running on my FreeNAS box, which was harder than the built-in plug-in version but was not horrible. Plus, if this goes well, it would be a one-time cost.

After the FreeBSD jail was set up, I opened a shell window into it and typed pkg install plexmediaserver which thankfully took care of downloading and installing all the binaries. The install process also ended with a few paragraphs of instructions on how to configure Plex to automatically start every time that instance of FreeBSD (in my case, the FreeBSD Jail) boots up. Copy-pasting the commands worked as advertised and everything was fairly painless.

I was then able to enjoy my home Plex media server for a few weeks. But there was one unknown: upgrades. I didn’t know how upgrades would work until I needed to perform one, so it’s just a waiting game for a new version of Plex to be released and see how things propagate. This week, the wait was over!

The first notification was from my home Plex media server. Apparently it checks for updates periodically and notifies me when one is available. After waiting a day or two, the update propagated to FreshPorts. Once that occurred, I opened a shell window into the FreeBSD jail running Plex and issued the following commands:

  • service plexmediaserver stop to halt the old running Plex.
  • pkg update to download the latest database of available packages.
  • pkg upgrade to perform updates of any new packages, including Plex.
  • service plexmediaserver start to start the new version of Plex.

I reconnect to the web interface of Plex running on my FreeNAS box, and I see the latest version up and running. Success!

SGVLUG: Custom Computer Projects

Last night I had the opportunity to present my Luggable PC, FreeNAS Box, and Portable External Monitor projects to the San Gabriel Valley Linux User’s Group. Though the projects themselves have only minimal relation to Linux, the spirit of customization and project sharing fits well with the Linux open source ethos.


I hauled in all the latest versions of my projects. Plus all the earlier drafts and revisions that have yet to be disassembled and pitched. More visual aids is always better than less and they proved quite popular after the talk concluded and people came up to look over the projects up close.

Some of the audience found the topic engaging and stayed after the talk discussing aspects that didn’t make it into the talk and offered ideas for future exploration. Some of those ideas were already on my to-do list and some are novel ideas I should explore.

A few people left early, whether they had other obligations or they got bored I might never know.

I don’t have a lot of public speaking experience so this was a great opportunity for me to get some practice in a low-pressure environment in front of a like-minded crowd. At the moment I’m not planning to go work in a mega corporation again. I might not need good presentation skills in a small business, but if I want to get entrepreneurial and start my own business, I will definitely need presentation skills.

This was good practice, building up the public speaking skill one bit at a time.

Much like my design and fabrication skills.


Acrylic Lights: Infinity Mirror

I’ve played with putting lights in my 3D-printed creations for glowing illumination effects. There were limits to what I could do with 3D printing, though, because printing with a clear filament does not result in a clear object. In contrast, acrylic is clear and works as a light guide with a lot of possibilities.

I’ve noticed a few attention-getting light effects in my acrylic projects to date, most of them created by happy accident. The acrylic box with external fixture made good use of external light. The Portable External Monitor version 2.0 was built from stacks of acrylic sheets: its fluorescent back light reflected between the layers like an infinity mirror.


This effect was on my exploration to-do list for the future, but I moved it to the top of the list after seeing surprisingly good results on the FreeNAS Box v2 enclosure.

I had planned for it to have the standard PC status LEDs: one for power, and one for disk activity. The acrylic plate for motherboard mounting spacer also had two cutouts for 3mm LEDs along the center line. The red hard drive activity light is to be mounted high, and the blue power light mounted down low. The idea was for the blue light to illuminate the top edge of the plate. When there is hard drive activity, red LED will light up the center of that edge, and it should blend to purple with the power light. Both LEDs were blocked from direct view by the motherboard, so all we should see is a nice soft glow emitting from behind the motherboard.


That was the plan, the reality was different. The red activity light worked as expected: when there is disk activity, the center of the top edge had a little red glow.

The blue LED decided to ignore my “nice soft glow” plan and put on an extravagant light show. It didn’t just light the top edge, it lit every edge of that acrylic sheet and had plenty of extra light energy to throw on the surrounding shelving.


Here’s a close-up of the sideways illumination.


The many rays visible in the side illumination, as well as the lines making up the top illumination, indicate infinity mirror action going on inside that sheet. It wasn’t directly visible, and probably very difficult to photograph even if so. Without internal reflections, the blue light would have just gone straight up. But with the smooth surfaces and edges of the acrylic reflecting inside the sheet, the light of a single LED bounced around, found different angles, and was emitted in many more directions.

This LED illumination effect warrants further investigation. It is a happy accident that I fully intend to learn from, and put into future acrylic projects.

I want every acrylic project to look this awesome!


A Survey of Hosting Mechanisms in FreeNAS

After getting Plex plug-in up and running, I started researching the FreeNAS features for hosting other code. I plan to keep my FreeNAS box up and running at all times, as is typical NAS usage, to ensure files are available whenever I want them.  I wanted to know what else I can run on the box at the same time since it is going to be on and consuming electricity anyway.

From highest and most user-friendly level to the lowest, they are:

1) FreeNAS Plug-in: This is how I got started with Plex media server, as it is one of the few plug-ins on the default list. Some flaws with this system were visible immediately. The version of Plex on the default plug-in library is several versions out of date, and there is no user-friendly update mechanism. The user has to go into the FreeBSD jail and update manually. Similarly, in order to access the media files hosted on the same FreeNAS box, the user has to know about manually mapping storage into FreeBSD jails.

It feels like the FreeNAS plug-in ecosystem never matured as much as the creators had hoped. A sentiment confirmed by this page. It explained the reasoning behind the push in FreeNAS 10 (Corral) to move to a system based on Docker containers. Unfortunately, when FreeNAS 10 was abandoned, that push was also put on hold.

Summary: It looks like FreeNAS plug-ins are a dead-end for a deprecated architecture.

2) FreeBSD Jail: Since a FreeNAS plug-in user basically had to know about running a FreeBSD Jail anyway, they might as well learn to work at this more hands-on level rather than depending on the FreeNAS plug-in architecture to sugar-coat it. There are a lot more steps involved, but for popular things, somebody would have posted the list of steps. For example, here’s how to install Plex in a jail. (UPDATE: I’ve learned Plex media server is part of the standard package system and therefore even easier to install: Create a jail, open shell to the jail, type ‘pkg install plexmediaserver‘, done.)

Upside: FreeBSD jails’ isolation protects FreeNAS from some security exploits. The resources consumed by a jail is managed by the same system that manages the rest of FreeNAS and automatically gains all the benefits thereof. Storage in a jail can be mapped to a FreeNAS storage volume to allow (optionally read-only) access.

Downside: FreeBSD jails offers no protection against the nastier kernel-level security exploits.

Summary: FreeBSD Jail makes sense for running relatively trustworthy code that integrates with the volumes on FreeNAS. Plex media server is a good example.

3) Virtual Machine: New in FreeNAS 11 is a feature to create and run virtual machines alongside FreeNAS using the FreeBSD bhyve hypervisor. As of FreeNAS-11-0-U1, this new feature is quite immature. For example, trying to stop a VM in the FreeNAS UI seems to have no effect. I have to go into the administrative shell and use the “bhyvectl‘ command-line utility to stop the VM. As another example, the virtual UEFI boot sequence doesn’t act as some operating systems expect, which can result in the user getting dumped into the UEFI shell. (Something normal users should never see!)

UEFI Shell

Google pointed me to this page which will help with most Linux distributions that encounter this problem. Thanks to this tip, I got my Ubuntu test server up and running on a FreeNAS VM.

Upside: Full virtual machine isolation will protect against most security exploits.

Downside: Full virtual machine isolation consumes more system resources. Most significantly, the storage is not shared: Space dedicated to a VM is not available to the rest of FreeNAS. And there is no storage mapping so a VM could only access FreeNAS shares over the network interface as if it is physically a different machine.

Summary: Virtual machines make sense for things that do not interact with the rest of FreeNAS, and a good alternative to setting up another physical machine.

FreeNAS Plugin: Plex Media Server

plex-logo-e1446990678679Once I had a few simple network file shares set up in FreeNAS, it was enough to do most of what I want in a home network storage device. For a fraction of the cost of a commercial solution like Drobo. Now we can start looking at the less critical fun stuff.

Part of my home media collection stored on my NAS includes various video files that I’ve been carrying around. Most of them were standard-definition video files I recorded off of broadcast television programs. This was done at a time when most people would record to VHS tape. Only the super nerdy types record to computer video.

So I had the files, but I didn’t have a good way to play them straight off a file share. This is where something like Plex comes in. There’s a server-side component that runs on my FreeNAS box, talking to client-side components for various devices. The web client could cast to a Google Chromecast, and the Amazon Fire TV stick has a Plex client app.

For security isolation, FreeNAS runs plugins inside a “jail”. This is a FreeBSD feature that sounds a lot like a Docker container but isn’t a Docker container. This isolation is good default security, but it does mean the Plex Media Server plugin could not see the rest of the FreeNAS box until the user specifies a way for the code inside the jail to see specifically allowed files outside of a jail. I could even specify the storage visibility to be read-only so there would be no accidental manipulation of my video files.

Once I got past the FreeBSD jail mechanics, it was mostly smooth sailing. The only problem came from the large fraction of my files encoded in Windows Media Video format, an old video format that Plex does not support. If I end up deciding I like the Plex experience, I will have to look into doing a bulk re-encode of these old video files.

FreeNAS File Sharing: Trust the Wizard

FreeNAS LogoThe authors of FreeNAS tries to make things easy for the user by providing automation tools (“Wizards”) that take care of the fine administrative details without requiring the user to learn all the underlying nuts and bolts of FreeNAS, or FreeBSD, or Linux kernel, etc.

This is especially true for creating network file shares in FreeNAS. It supports many network file sharing protocols. Including the Apple-specific AFP (Apple File Protocol), the Microsoft Windows-based SMB (Server Message Block), the Unix-based NFS (Network File System), plus three others I don’t even understand.

Each of these have their own setup requirements that a casual user like myself is unlikely to get right on our own. So the manual encourages the use of Wizards. (Bold emphasis mine, but I think it should be in the manual!)

FreeNAS® provides a Wizard for creating shares. The Wizard automatically creates the correct type of dataset and permissions for the type of share, sets the default permissions for the share type, and starts the service needed by the share. It is recommended to use the Wizard to create shares.

Windows has a file sharing wizard as well: In Windows file explorer, I would create the folder I want to share, then right-click on that folder to select “Sharing…” This launches the wizard who will then take care of everything else.

Since I was used to the above workflow, I did the same thing in FreeNAS. Create a directory, then run the wizard to share that directory. Unfortunately this results in network shares that were not accessible. (“Access Denied”)

I eventually debugged the problem to my “create a directory” step. Since I had created as an administrator, the permissions on that directory were not set for use by other users. And the FreeNAS wizard did not (or could not) update the permissions properly.

What I needed to do was to launch the FreeNAS network sharing wizard, and tell it to create the directory as part of the network share creation process. This way the directory would be created by the wizard who will properly set the permissions for file sharing.

I did too much and that became unhelpful.

Trust the Wizard.

FreeNAS USB Flash Boot Drives: Recovering Boot Drives That Don’t Boot.

FreeNAS LogoWe’ve established that FreeNAS can mirror the USB flash drive boot device for redundancy. If one of them should fail, the system can be recovered with the other. I hadn’t set out to verify the recovery procedure, but I stumbled into a practice run anyway. In hindsight it’s good to get a feel of my recovery options now when I’m still playing with the system, rather than later when I’m in a panic to recover my data.

Over the past few days I’ve experimented with the boot volume, setting up the mirroring and using the built-in replacement mechanism to retire my USB sticks with checksum errors. After this was done, I unplugged the checksum-error drives (including the one I originally installed FreeNAS on) and rebooted the system.

It failed to boot and ended up at the GRUB recovery menu. Hmm, that’s not good.

I plugged all the old drives back in… and that didn’t improve the situation. Apparently, in the midst of all the mirroring and replacing, I managed to damage GRUB configuration so FreeNAS no longer boots.

Since I expected all the actual operating system files to be OK, I searched for a way to rebuild just the boot loader portion of the USB sticks. There are many ways to do this. The easiest – looking route I took is to perform an in-place upgrade.

I booted up the FreeNAS installation media, and selected the “Install/Upgrade” option. I pointed it to a USB drive that was free of checksum errors but would not boot. The installer detected an existing installation and offered to generate a new boot environment for the existing FreeNAS installation. Sounds perfect! I hit <OK>

Ten minutes later, the upgrade failed due to an error writing boot configuration files the installer wants to add. A quick Google search found a few suggestions on how to fix it using command-line tools I wasn’t familiar with. I decided to try easier things first.

I restarted the installation process and chose the other upgrade option: An upgrade with a disk reformat rather than just generating a new boot environment. The installer will preserve a copy of the FreeNAS configuration but wipe everything else from scratch.

This approached was more destructive and took more time but it worked. My FreeNAS box booted back up as if nothing happened. Success!

FreeNAS USB Flash Boot Drives: Wide Variations in Performance

Since I went into this FreeNAS project with the expectation to experiment and learn, I followed the recommendation to use commodity USB flash drives as the operating system boot drive despite my skepticism. The previous blog post discussed checksum errors found on some of the USB flash drives I had on hand, and how FreeNAS is able to mitigate the errors by mirroring the operating system across multiple USB sticks. This greatly increased my confidence entrusting the FreeNAS operating system to these USB thumb drives.

Usually these devices are used for ferrying files from one computer to another. They expect to see some files copied onto the drive, then copied off the drive in order. (sequential read/write operations.) Putting them in service as the operating system drive is an entirely different work pattern, with small pieces of data written at unpredictable places and other data retrieved from equally unpredictable places. (random read/write operations.)

I was aware of this difference and it was part of my skepticism using USB sticks as operating system drive. Now that I’m using it, I can see how it works in reality. Thanks to FreeNAS boot device mirroring, I have the confidence to go into this experiment without risking my data.

Outside of errors or outright failures, the other thing I had expected was degraded performance. If the flash memory controller chip is optimized for sequential operations, it might be bad at random operations. Unfortunately there was no way to tell up front. The software running on the flash drive controller isn’t something manufacturers put on the packaging. And every company has their own algorithm.

Since they are so cheap, the easiest thing to do is to just go and try it. The result is clear: some USB flash memory drive controllers are far far better at this workload than others.

Some of this difference is felt immediately, in the time taken for the mirroring duplication process. During this “resilvering” process roughly 2GB of data is copied to the new flash drive. Out of the two drives that had no checksum errors, one of them was able to complete the process in a little over two hours. The other took 26 hours, more than ten times longer!

Their difference is also visible in the FreeNAS system performance reporting page. Since FreeNAS keeps the two drives in sync, it’s easy to see how they respond to identical workloads. The flash drive identified as (da1) breezes through work, never more than 50% busy. Its mirrored sibling (da2) struggles to complete the same work, frequently spiking up to 100% busy.


One of these is much less happy at their new job than the other.

Between the checksum errors and the disk busy graph, I am now much better enlightened on the varying quality of USB flash drives. I had thought of them as all basically equivalent, so I just bought whichever is the cheapest. My attitude has now changed. From here on out, whenever I need to buy USB flash drives, I’ll look for those made by SanDisk.

FreeNAS USB Flash Boot Drives: Mirroring For Fault Tolerance.

FreeNAS LogoFreeNAS encourages the use of USB flash drives as the operating system boot drive. This allows FreeNAS to dedicate all of the motherboard SATA connectors for data storage drives. I didn’t think commodity USB flash drives are trustworthy enough to hold the operating system, but I was willing to experiment and be proven wrong.

The very first night, I got worrying news from the nightly system check:

  pool: freenas-boot
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 3K in 0h10m with 0 errors

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0    11

errors: No known data errors

Looking on the bright side, “No known data errors” is comforting, as is the “repaired […] with 0 errors”. It’s nice FreeNAS was able to repair whatever was wrong with my USB stick. I suspect inexpensive commodity USB flash drives frequently encounter errors that are silently corrected by the operating system. Still, an error is an error and it’ll only be a matter of time before I run into a serious problem.

Fortunately, FreeNAS authors had the foresight to make sure a bad boot device does not become a single point of failure. A second one can be added to the system act as a mirror to the boot device. If either of them fails, the other can take over.

Much to my dismay, the second USB stick I tried also encountered a data checksum error. I didn’t have much luck figuring out how to interpret the checksum error code, but I did learn that it is supposed to be zero. The first stick returned 21, the second 26.

I tried a third USB stick and was relieved to finally see a zero checksum. The output below was generated when I ran ‘zpool status’ while the third stick is in the middle of replacing the second stick.

  pool: freenas-boot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress

        NAME             STATE     READ WRITE CKSUM
        freenas-boot     ONLINE       0     0     0
          mirror-0       ONLINE       0     0    21
            da0p2        ONLINE       0     0    21
            replacing-1  ONLINE       0     0     0
              da1p2      ONLINE       0     0    26
              da2p2      ONLINE       0     0     0  (resilvering)

errors: No known data errors

I also found a fourth USB stick that was checksum error-free, so I had it take the place of the first one.

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h29m with 0 errors

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0    21
            da1p2   ONLINE       0     0     0
            da2p2   ONLINE       0     0     0

errors: No known data errors        

Now both boot drives in the mirror set have zero checksum error, but the mirror volume overall still has checksum error 21 from the first USB stick. I’m still learning if that means anything (bad) and what it would take to reset that to zero.

FreeNAS Box v2: Component Access

One aspect that was completely neglected in v1 was any kind of an access door. The panels were designed only enough for them to mesh, so the only way to hold things together is to glue everything shut. Completely impractical! The only way to perform any maintenance would be to shatter some acrylic to break the case open.

Now we have two access panels, one in the front and one on the bottom. I tried to see if I could somehow integrate things so there would only need to be one access panel, but never came up with a design that would work well while simultaneously satisfying all the other design goals I’m trying to accomplish.

Each access panel is held in by 4 x #6-32 screws, the standard desktop PC case screw. They fasten into threads I had tapped into the underlying layer of plastic. I should use heat-set inserts for better durability. Unfortunately I didn’t have #6-32 inserts on hand at the time and I thought I would probably build a V3 follow-up anyway.

FreeNAS v2 Front Panel Open

The front panel opens to allow access to the front chamber where the motherboard and CPU lives. If I wanted to upgrade the memory or switch out the SATA cables, I could do so through the front panel opening.

The bottom panel opens to allow access to the rear chamber. The two hard drives and the power supply are all installed through the bottom opening. The drives can be individually replaced without fighting through too many pieces of cable. The power supply can also be replaced through the opening exposed by the bottom panel.

FreeNASv2 Both Panel Open

FreeNAS Box v1 to v2 Size Comparison

Now that FreeNAS Box v2 is up and running, let’s do a size comparison to see how things have changed. The width dimension was a regression: v2 is wider by 3 centimeters. The real space requirement increased even more than that, because the v2 air intakes are on the sides.  So it needs additional room to the left and right of it in order to avoid blocking those intakes. In contrast, v1 with its bottom intake would be OK sitting flush against objects to its left and right.

Looking at the final results in reality, I think I can rearrange a few things to reduce the width by 1 cm. Something to consider for a potential FreeNAS Box v3.

FreeNAS v1v2 SideBySide

The greatest improvement is in the height which has been reduced by 7 cm thanks to the elimination of the lower air intake cavity. The advantage is somewhat reduced when in use, because the power cable now sticks straight up and adds a bit to the required height. If this becomes a serious problem, though, I could always switch to an power cable with a right-angle plug. This will allow the box to fit in a shelf only 28 cm high, leaving barely enough for proper heat exhaust. Looking over the results, I think I can safely reduce the bottom of the case by 1 cm and everything will still fit, another change for the potential v3.

FreeNAS v1v2 SideBySide

And finally, the depth is reduced by 3.5 cm. Real world improvement is even more, because now the back of the box can sit flush against the wall in a way that v1 could not.

The reduced height and depth has more than compensated for the increase in width, v2 is overall a more compact space-efficient package than the v1 design.

FreeNAS Box v2: Construction Complete

After spending an afternoon + evening at a Tux-Lab work session, I have my FreeNAS Box v2! It’s always fun to see my idea turned into reality.

FreeNASv2 Complete

The first thing I appreciated was the fact that the components are clearly visible through the acrylic panels. And even better, the messy tangle of wires are hidden behind them. This reversal from v1 is the best aesthetic change.

The other major design requirement – that both cooling fans be visibly spinning – is also present but it doesn’t have as much of an immediate aesthetic effect.

After I’ve kept it on and running overnight, I checked the temperature around the box the following morning. I think it would be neat to check thermal performance with a FLIR thermal imaging camera but lacking such toys I went with the low-tech way of putting my hands at various places around the box to feel the temperature.

The front chamber – where the CPU and motherboard reside – has a slight temperature gradient from top to bottom but overall it was relatively cool to the touch. This was expected as the CPU basically sat idle all night. It also means I won’t need to cut a hole in the front door for a direct air intake.

The rear chamber, with the power supply and both hard drives, is where most of the heat is generated. The two drives were warm to the touch signifying that they’ve been spinning all night and getting some amount of cooling to keep them from getting hot.

The power supply fan was running and the power supply case was cool to the touch. The power meter read 30W for the FreeNAS box in this steady-state idle state. This is a very light load for a 600W-rated power supply, reflected in its cool running temperature.

FreeNAS Box v2: Construction Fixture

One of the problems with FreeNAS Box v1 was that I designed it with tabs and slots to fit into each other. While it made the box easy to assemble, the slots severely weakened the structure of the box.

For FreeNAS Box v2 I avoid the tabs and slots. But I need something else in their absence to help me during construction. The answer is a fixture: Something I design along with the box that helps me build it, but not part of the end product.

Building the box will start with bonding all the major vertical pieces together. Once the cement has set hard enough for them to stand alone, the fixture pieces can be removed. The resulting assembly will then be self-supporting as the remaining pieces are attached.

The fixture pieces sit top and bottom. Pretty much where the largest horizontal pieces would eventually go, but are distinctly different from those pieces.

FreeNASv2 Fixtures.JPG
Initial assembly (gray) with assembly fixture (yellow)

The top fixture has two slots for holding two of the vertical sheets of acrylic. We’ve already established such slots are bad for the structural strength of the end product, but it’s perfectly OK (and quite useful) to have them in a fixture.

Both the top and bottom fixture have round cutouts in the corners and in the mid-span T-joint so that they stay clear of any extraneous acrylic cement that might leak out. This way we avoid accidentally cementing the fixture to the product.

Each of the fixture is made of two layers of acrylic, a main layer and a secondary layer whose shapes helps keep the box pieces in place. The small round circles visible in the picture is sized for M4 screws to fasten the fixture layers together. Using screws instead of acrylic cement allows us to later disassemble the fixture and recycle the pieces as scrap acrylic in future projects.

FreeNAS Box v2: Airflow Design

Designing the system airflow for thermal management is a huge consideration for the FreeNAS box design. The two fans in the system have been oriented for easy inspection first and then the airflow was designed second to work with natural convection flow. Lacking skills to use sophisticated thermal modeling and analysis tools, this design is mostly based on intuition.

Working as a network attached storage device is not a very computationally intensive task. Plus the CPU has its own fan, so thermal control of the processor is not a primary concern. The power supply also has its own fan, which I assume can take care of itself.

That leaves the hard drives as the primary thermal concern. Lacking their own cooling fans, the airflow design of the case will put them first in line to receive the coolest air. This meant placing them right at the intake. The v1 intake was on the bottom, so that’s where the drives were. The v2 intake air from the side, and again that’s where the drives were placed.

After the intake air has met the hard drive, the warmer bits should flow up and over the top of the hard drive, carried by convection towards exhaust. The cooler bits should head towards the rest of the case, helping to cool the motherboard and the CPU.

The motherboard and the CPU is in its own chamber. Cold air comes in the bottom and sides of this chamber and the top of this chamber has holes to send its warmest air into the power supply fan for exhaust.

If this proves to be inadequate cooling for the motherboard, we have the option of cutting an air intake hole directly in the front door panel of the case.  The CPU fan can then pull in cool air from the outside. This will reduce the amount of air drawn in past the hard drives, though, so I wanted to see how well it works before I start cutting holes.


FreeNASv2 Thermal Flow 2

FreeNAS Box v2: Additional Goals

We’ve just established all the problems exposed by the v1 prototype that we want to address for v2. In addition to those issues, we also want v2 to cover a few things that are no fault of the v1 prototype.

First one is relatively obvious: actually build a complete and usable case. I knew I was trying new ideas in v1 and that something will go wrong, so it wasn’t really complete. Even if everything went right (I knew it wouldn’t) I would have had to build a v2. For one example, I didn’t bother to design an access door.

We then have a few separate items that relate to improving space efficiency.

When I placed v1 on the shelf where I expected to keep my FreeNAS box, it wasted a few inches between the back and the wall due to the angle of the power cable. I want to rearrange things so that the back of the box can sit flat against the wall.

Cooling path in v1 started with air intakes on the bottom of the case. This was part of the tribute to the Apple PowerMac G4 cube, but functionally unnecessary while consuming vertical space.

Also contributing to the vertical space consumption was pointing all the ports downwards, like the PowerMac G4 cube. This made the ports difficult to access. It would be good to align the direction of all the plugs, so power and network cables can be in parallel.

Out of all the requirements listed here and in the previous post, the greatest impact was the “make sure all fans are visible for easy verification they are spinning” goal. It meant rearranging the components so both fans face forward. This made for an interesting design challenge as it is against common convention of computer case design. Once I got that set up, the configuration was then further refined in Fusion 360 to satisfy all the remaining requirements until we have this: my FreeNAS Box v2.


FreeNAS Box v1 Problems

FreeNAS Box v1 was a good learning project for acrylic construction. Here are the issues with v1 I want to address for the next version.

  1. Non-orthogonal joints: The laser cutter only cuts right angle edges. v1 had a few joints that were impossible to cement because the edges didn’t align at right angles.
  2. Tab and slot construction: To help align joints, I designed v1 with tabs to fit into slots I had also cut into their mating surfaces. While this made the box easy to build, it destroyed durability of the end product. The sharp corners of the slots are where acrylic starts cracking under stress. I had known about the dangers of sharp internal corners, but I thought acrylic cement would bond everything together eliminating the weak point. This idea has now been proven false.

    Stress cracks that started at corners of slots.
  3. Unappealing tangle of wires: The v1 box design placed all of the wires up front, which turned out to look pretty ugly, and all the components (hard drives and the motherboard) were hidden under the mess.

    Yes, there’s a computer under the tangle of wires.
  4. Difficult access to components: Besides looking bad, the mess of wires up front also blocks access to everything else. It would be difficult to perform maintenance such as replacing drives if they fail.
  5. Cooling fans vulnerable to jamming: The wiring paths were such that, if some wires should misbehave and bend slightly out of position, they would impinge upon the blades of cooling fans stopping them from turning.

    Several wires in this bundle poked into the fan grill where it does not belong.
  6. Cooling fans are out of sight: Compounding the problem of blocked fans is the fact that despite the clear acrylic exterior, it was not easy to notice the fans were blocked.

I had to physically build FreeNAS Box v1 before I knew known any of the above are problems. Some I had thought about and didn’t think would be a problem, the others I just hadn’t thought about at all.