Portable External Monitor v3: Trim Unused Bracket

For the portable external monitor project, version 3 (PEMv3) , we want to pack our components more tightly together to create a compact package. The most inconvenient barrier to the goal is the old back light driver board mounting bracket. While the circuit board itself was already dead when I got the laptop and promptly removed, the bracket for the driver board was still present during PEMv1 and PEMv2. The aftermarket back light driver board is significantly larger than the original Dell part and would not fit in the bracket. I had originally hoped the bracket can be recruited into a new role but none had ever come up.

So now it is time to trim off the bracket.

The front part of the bracket is riveted to the thin metal bezel of the LCD panel. The rivet makes it easy to remove with a Dremel grinding tool, but I was concerned about the potential for metal particles to damage the panel. I spent more time covering and wrapping the rest of the panel than I did removing the rivets.

LCD bezel rivet

The rear part of the bracket is part of the panel mounting frame. Since there were no electronics here, I didn’t have to worry as much about damage. I will need to thoroughly clean up the work piece afterwards to minimize the number of particles that are still sticking, but I didn’t have as much risk while doing the actual cutting.

Frame Before

I chose to minimize the cut length which removed more metal than necessary because the shortest cut connected the two large bottom holes. We should still have enough structural strength and mounting holes for the frame to be useful.

Frame After

With the old bracket out of the way, it’s time to pack our components into the space previously occupied by the bracket.

 

Portable External Monitor v2 Problems To Fix in v3

And now, back to the portable external monitor (PEM) project. Version 2 has been in use for several weeks, called into duty when I needed a screen. It was used to help me install and smoke test ROS on a Raspberry Pi 3. It was also used for the FreeNAS box, which was ideal because FreeNAS only needs a screen briefly for setup.

Functionally speaking the portable external monitor has been working well. But the physical form factor left room for improvement: it wasn’t as portable as I would like.

The first part of the problem is weight: PEMv2 was built by stacking sheets of acrylic that had holes cut out to make room for components. There’s no fundamental reason why this subtractive construction technique had to be heavy. But in practice, holes were cut only when needed to make room. The result is a lot of excess acrylic. Acrylic is not super heavy, but it all adds up to just over 8 pounds which is ridiculously overweight for a 15″ screen.

The second part of the problem is size: PEMv2 was intended to fit within the width and height of my JanSport backpack. When assembled, I had a facepalm moment: it does not actually fit into the backpack because PEMv2 is rectangular and the backpack is not. The top of the backpack is rounded, with no room for PEMv2 top corners.

BackpackPEMv2

Lastly: while PEMv2 was thick enough to stand upright on its edge, it is easily tipped over and the upright vertical screen angle is not very ergonomic. We can definitely do better.

PEMv3 will try to address all of the above issues. To minimize physical volume, we’ll need a more compact way to package all the components. We’ll need to build our enclosure using less acrylic to further reduce weight. And we’d like to have an integrated stand that can securely prop up the screen at a better viewing angle

Broken Source is not Open Source

After attending Elecia White’s “On Cats and Typing” talk, I felt a little more motivated to look into robots. The robot arm used in her demo was the MeArm by Mime Industries. It is built out of commodity micro servos and laser-cut acrylic. I looked at it and thought I could get one up and running on my own. It is sold as a self-assembled kit but in the spirit of open source, people are also allowed to laser-cut their own pieces using the open-sourced DXF file available via Github.

Or at least that was the theory.

In practice, the DXF doesn’t work. Inkscape couldn’t load it. CorelDRAW couldn’t load it. Onshape couldn’t load it. Fusion 360 – from Autodesk, the people who created AutoCAD which is where DXF came from – couldn’t load it.

MeArm Broken

Well, that was disappointing.

Google results confirm I’m not alone. I found many reports of people failing to get this DXF to work for them, and not a single success story. Of course, there’s a little selection bias here: people who encounter no problems rarely go on the internet to announce they had no problems. But I would have expected a few of the forum posts from people having problems to get some positive responses, and I didn’t find any of those.

This is frustrating. I’m unlikely to go and buy the kit when I already had most of the pieces on hand and it is in theory open source for me to make my own. It’s impossible to tell if there’s a perfectly innocent explanation or if this was done maliciously to slap on the “open source” label without actually risking any cut into sales. Whatever the explanation for why this DXF is broken, it doesn’t change the fact that it is broken. When the publicly available source file is unusable. Is it still open source?

I vote no.

Building a Lithium Ion Battery Pack with S-8254A Protection IC

Encouraged by my earlier success buying somebody’s implementation of an IC reference design off Amazon, I went looking for more. The previous project used a MP1584 chip to regulate the variable voltage of a battery to a constant level for a Raspberry Pi 3. Now I want to improve upon the battery side.

The battery pack had three 18650-sized lithium-ion battery cells in series. These cells were salvaged from an old Dell laptop’s power pack. Since they were ten years old, there’s a bit of a question mark hovering over them. A battery pack that simply wires them in series risks over-charging or over-discharging the weakest cell. This abuse of lithium-ion cells usually ends in a fire.

I searched for a battery management circuit board to help me avoid setting my projects on fire. I settled on this item which is built around the Seiko Instruments’ S-8254A IC. This circuit board will monitor individual cells. If any of the voltage levels exceed safe limits for lithium-ion cells during either charging of discharging, the chip will disconnect the whole pack.

Once everything was connected according to instructions, I have a battery pack that I can use with much higher confidence.

BatteryManager

Using my Astro-Flight power meter, I put this battery pack through a full charge and discharge cycle. Something I was squeamish to do before the battery protection board. Upon completion of the cycle, the power meter counted 1.65 amp-hours. The text printed on the cells say LGDA2E18650, which had a nominal 2.25 amp-hour capacity when new. Ten years old and 73% of nominal capacity is not bad, and perfectly usable for a wide range of future projects.

Powering the Raspberry Pi 3 With MP1584 Voltage Step-Down Converter

The Raspberry Pi 3 is a very impressive piece of hardware for the price, but it has its flaws. One challenge is supplying power to a Pi 3. Like all the Pi boards, power is supplied via a standard micro USB plug. This implies the Pi only needs USB level power with its specified maximum of 0.5 amp @ 5 volts = 2.5 watts. In reality, this USB port is abused beyond the specified range. The Raspberry Pi foundation recommends the power supply for a Pi 3 should supply up to 2.5 amp @ 5 volt = 12.5 watts. Five times the USB specification maximum.

None of the USB power sources I already had could handle this workload. I originally had ideas about running a Pi 3 off of a portable USB charger, but that failed under the vastly greater power draw. I went looking online for solutions.

I needed an efficient DC to DC voltage regulator that can handle the maximum power draw of a Raspberry Pi 3 without consuming a lot of power itself. Since the voltage of a battery changes as it drains, the converter needs to handle a range of input voltages while holding the output voltage steady.

The MP1584 chip from Monolithic Power Systems fit the bill, but I didn’t want to deal with a tiny surface mount IC, nor do I have the skill to design the supporting circuit required. Consulting with a few electronics hardware hobbyists, I got the recommendation to take the reference design out of the datasheet and build that.

And then, an even better recommendation: If it’s a popular chip, and its reference design is good enough, somebody would have mass-produced it and put it on Amazon. And indeed, they have. A lot of vendors, in fact, from all around the world.

I scrolled through a few of the listings but didn’t really have a good feel on how to judge one vendor against another. So I took the easy way out: I clicked on the “Amazon’s Choice” link to this offering.

Once the module arrived, I soldered battery connectors to the input and a micro USB plug to the output. I adjusted the output voltage to 5 volts, and connected everything to power up my Raspberry Pi 3.

PiVoltage

So far it has worked very well. The Raspberry Pi 3 stayed running through tasks that demanded extra power, that would previously trigger a low-power brownout with my existing USB power sources. The output voltage held steady as the battery drained.

Functional, inexpensive, and I didn’t even have to deal with surface mount components! This was a win.

 

Simple Circuit Board On 3D-Printed Plastic

CircuitBoardHere’s a behind-the-scenes follow-up to the LED test fixtures of the previous few posts: when we only need a simple circuit for a 3D-printed project, we can meld the two instead of using a formal circuit board. In this context “meld” is meant literally: the parts of the circuit can be heated up with the soldering iron so they melt into the 3D printer plastic.

When I built the dual-LED acrylic illumination test rig, I wanted the simplest circuit possible. It’s not something I need to be durable long-term and I wanted to be up and running with my tests as quickly as possible. The full length of a resistor and its wires are almost long enough to bridge the gap between the two sides of the fixture, so I tried to make that work.

When I started soldering all the wires together, I had planned to just leave everything dangling. But the close proximity of the soldering gun to the 3D printed PLA plastic started softening the plastic and I realized I can use this to my advantage. A few seconds with the soldering iron was all it took to heat up a wire so it can be melted into and embedded into the plastic. The resistors themselves took a little more effort, but I sunk them into the plastic as well. The LEDs had been held in place by their bent legs, which was sufficiently stable but had a tiny bit of wobble. Melting the plastic around LED legs gave us a much more secure placement.

Components melted into the plastic are no longer subject to flexing and eventually breaking from metal fatigue. Add a strip of electrical tape to guard against short circuiting to complete the quick and simple circuit to light up the test rig LEDs.

Illuminate Acrylic Edge: Test Fixture 2

After running through a few acrylic test pieces looking for the best edge illumination, I decided I need a dual fixture to allow side-by-side comparison as I swap through test pieces.

IMG_5158

Another change I made in the text fixture is to remove the aluminum foil at the bottom. While the foil may be useful to direct light, it distracts from the testing. If a particular test piece is losing light to the fixture, I don’t want that light reflected back in. I want to be able to see the failure in the form of illuminated white plastic. When there are no acrylic test pieces in the fixture, the cone of illumination is clearly visible.

IMG_5159
Test fixture #2 illuminated without acrylic test pieces.

The two sides aren’t exactly identical. One of the LED is slightly brighter than the other, and the two sides ended up with slightly different textures. But it should be good enough for our comparison purposes.

The first fixture implied that the cavity surround the LED is where we should focus our attention, so let’s try a few shapes. A square and a circle seems to differ only slightly in the brightness of the center top hot spot.

IMG_5161
Square LED cavity (left) and circular LED cavity (right)

A triangular cavity was much more interesting – all the light has been diverted from the top center, sending them off to the side. And I tried a teardrop shape just to see what would happen. The important detail to note on the teardrop is that a lot of light was lost to the fixture instead of being sent to the edges. This tells us the cavity edges should be as small as possible to push its surface right up against the LED to reduce light loss.

IMG_5162
Triangular (left) and teardrop (right)

The cavity sizes were then minimized for the next set, again testing for different shapes. A flat top to the cavity didn’t work as well as the cavity shape conforming to the LED shape.

IMG_5163
Flat top cavity (left) and conforming curve cavity (right)

But the best results came from putting a small curve in front of the point of the LED. This appears to break up the central beam and sends it to the edges like we want.

LED scatter curve

From an cost/benefit ratio perspective, this small curve is a winner. It is a very minor change to the geometry and yet it delivers significant improvement to the resulting light. When put into a larger sheet of acrylic, with greater number of internal reflections, it should do quite well. And for a little extra smoothness in illumination, we can take a piece of sandpaper and lightly roughen up the surface. Adding a frosted edge reduces the reflections somewhat, but it does help even out the overall illumination.

IMG_5165
Best illumination to date with the small curve (left) which can be further enhanced by a frosted edge (right)

These experiments have been quite informative. I look forward to applying what was learned here to future acrylic projects.

Illuminate Acrylic Edge: Goals and Test Fixture

After the surprising success of LED illumination in FreeNAS v2 enclosure, I wanted to spend some time experimenting with the concept. When searching for “acrylic edge illumination” on Google, everybody seems to be talking about positioning the LED at the edge of the acrylic sheet and lighting up the pattern of something engraved on the acrylic. My goal is the opposite: I want to place the LED in the middle of the acrylic, and I want the light to shine out to the edge of the acrylic sheet.

We start with the assumption that by default, a LED shining inside a piece of acrylic will only illuminate in the direction it is pointed.

Hotspot

Our ideal goal is to determine how to direct this light so it illuminates all the edges of the acrylic sheet, not just the direction of the LED face.

Ideal

I 3D printed a small test fixture for these experiments. It has space for a 75mm x 75mm test piece of acrylic and a LED that pokes up in the middle of that space. There’s a 10mm wide border around the test piece so I can observe the pattern of illumination beyond the edge. At the outside edge of the border, a wall to observe the intensity of illumination beyond the edge. A piece of black tape covers the direction of the viewer so the LED doesn’t overwhelm the rest of the observation. A piece of aluminum foil lines the bottom of the test fixture to reflect any light back into the acrylic.

IMG_5146

The fixture lights up as expected in the absence of any acrylic.

IMG_5147

These two experiments tested cutting grooves in the acrylic. One set had straight grooves, a second set curved. They were successful in breaking up the center top hot spot, sending some of that light elsewhere. But the light seems too concentrated on the bottom third.

IMG_5151

IMG_5152

Instead of cutting grooves, this piece tested cutting entirely through the acrylic. The circular shape does seem to disperse the light fairly well.

IMG_5149

These were interesting, but the most surprising result came from a test piece of acrylic with nothing cut in the middle. I had expected the light pattern to resemble the triangular hot spot of the LED by itself without any acrylic, but we got this:

IMG_5148

It has the same basic trend of the other light patterns in this set of experiments, which tells us the majority of the light scattering is not done by the curved/straight grooves or the circle. The feature with the largest impact is actually the small cavity surrounding the LED itself.

The fixture has been informative, but it has one problem: it is difficult to make comparisons between different test acrylic pieces. Before proceeding with investigation, the test fixture will be expanded so there are two test pieces side by side for comparison.

Acrylic Lights: Infinity Mirror

I’ve played with putting lights in my 3D-printed creations for glowing illumination effects. There were limits to what I could do with 3D printing, though, because printing with a clear filament does not result in a clear object. In contrast, acrylic is clear and works as a light guide with a lot of possibilities.

I’ve noticed a few attention-getting light effects in my acrylic projects to date, most of them created by happy accident. The acrylic box with external fixture made good use of external light. The Portable External Monitor version 2.0 was built from stacks of acrylic sheets: its fluorescent back light reflected between the layers like an infinity mirror.

PEMv2_InfLights

This effect was on my exploration to-do list for the future, but I moved it to the top of the list after seeing surprisingly good results on the FreeNAS Box v2 enclosure.

I had planned for it to have the standard PC status LEDs: one for power, and one for disk activity. The acrylic plate for motherboard mounting spacer also had two cutouts for 3mm LEDs along the center line. The red hard drive activity light is to be mounted high, and the blue power light mounted down low. The idea was for the blue light to illuminate the top edge of the plate. When there is hard drive activity, red LED will light up the center of that edge, and it should blend to purple with the power light. Both LEDs were blocked from direct view by the motherboard, so all we should see is a nice soft glow emitting from behind the motherboard.

FreeNASv2LightPlan

That was the plan, the reality was different. The red activity light worked as expected: when there is disk activity, the center of the top edge had a little red glow.

The blue LED decided to ignore my “nice soft glow” plan and put on an extravagant light show. It didn’t just light the top edge, it lit every edge of that acrylic sheet and had plenty of extra light energy to throw on the surrounding shelving.

FreeNASv2_LightsAbove

Here’s a close-up of the sideways illumination.

FreeNASv2_LightsSide

The many rays visible in the side illumination, as well as the lines making up the top illumination, indicate infinity mirror action going on inside that sheet. It wasn’t directly visible, and probably very difficult to photograph even if so. Without internal reflections, the blue light would have just gone straight up. But with the smooth surfaces and edges of the acrylic reflecting inside the sheet, the light of a single LED bounced around, found different angles, and was emitted in many more directions.

This LED illumination effect warrants further investigation. It is a happy accident that I fully intend to learn from, and put into future acrylic projects.

I want every acrylic project to look this awesome!

 

Luggable PC Wireless Module Installation

The point of the Luggable PC project is to build a mobile computer out of commodity desktop parts instead of more expensive specialized laptop parts. The upsides come from component choice, ability to upgrade piecemeal, and customization. One downside is that desktop components won’t have some of the parts taken for granted on modern laptops. Like today’s topic: wireless Ethernet.

The motherboard I currently have in the Luggable PC chassis (Intel DH87MC) does not have wireless Ethernet out of the box. I had been using a small USB wireless network dongle to provide wireless connectivity. The compact size is handy, but the compact size also restricts the antenna size, which in turn restricts performance.

The driver for the Realtek device isn’t anything to cheer about, either. It works OK in Windows, but it frequently fails in Ubuntu. I would frequently find myself without network access in Ubuntu and have to reset the USB adapter by unplugging it and plugging it back in.

I knew that my motherboard had a mini card slot for a wireless card. I also knew I had salvaged wireless cards from laptops I had disassembled for parts. But it wasn’t until today that I finally got around to plugging a wireless module into the Luggable PC.

I had also salvaged the matching antenna modules from the laptop. They formerly resided in the laptop lid, and now they are taped to the inside of the 3D printed enclosure.

Intel Wireless Card

Thanks to these two large(r) antennae, I now have stronger wireless signal and better data throughput. And the driver for this Intel-made wireless module has been far more reliable. And on top of all that, I’ve freed up a USB port.

One win for salvaged parts!

A Survey of Hosting Mechanisms in FreeNAS

After getting Plex plug-in up and running, I started researching the FreeNAS features for hosting other code. I plan to keep my FreeNAS box up and running at all times, as is typical NAS usage, to ensure files are available whenever I want them.  I wanted to know what else I can run on the box at the same time since it is going to be on and consuming electricity anyway.

From highest and most user-friendly level to the lowest, they are:

1) FreeNAS Plug-in: This is how I got started with Plex media server, as it is one of the few plug-ins on the default list. Some flaws with this system were visible immediately. The version of Plex on the default plug-in library is several versions out of date, and there is no user-friendly update mechanism. The user has to go into the FreeBSD jail and update manually. Similarly, in order to access the media files hosted on the same FreeNAS box, the user has to know about manually mapping storage into FreeBSD jails.

It feels like the FreeNAS plug-in ecosystem never matured as much as the creators had hoped. A sentiment confirmed by this page. It explained the reasoning behind the push in FreeNAS 10 (Corral) to move to a system based on Docker containers. Unfortunately, when FreeNAS 10 was abandoned, that push was also put on hold.

Summary: It looks like FreeNAS plug-ins are a dead-end for a deprecated architecture.

2) FreeBSD Jail: Since a FreeNAS plug-in user basically had to know about running a FreeBSD Jail anyway, they might as well learn to work at this more hands-on level rather than depending on the FreeNAS plug-in architecture to sugar-coat it. There are a lot more steps involved, but for popular things, somebody would have posted the list of steps. For example, here’s how to install Plex in a jail. (UPDATE: I’ve learned Plex media server is part of the standard package system and therefore even easier to install: Create a jail, open shell to the jail, type ‘pkg install plexmediaserver‘, done.)

Upside: FreeBSD jails’ isolation protects FreeNAS from some security exploits. The resources consumed by a jail is managed by the same system that manages the rest of FreeNAS and automatically gains all the benefits thereof. Storage in a jail can be mapped to a FreeNAS storage volume to allow (optionally read-only) access.

Downside: FreeBSD jails offers no protection against the nastier kernel-level security exploits.

Summary: FreeBSD Jail makes sense for running relatively trustworthy code that integrates with the volumes on FreeNAS. Plex media server is a good example.

3) Virtual Machine: New in FreeNAS 11 is a feature to create and run virtual machines alongside FreeNAS using the FreeBSD bhyve hypervisor. As of FreeNAS-11-0-U1, this new feature is quite immature. For example, trying to stop a VM in the FreeNAS UI seems to have no effect. I have to go into the administrative shell and use the “bhyvectl‘ command-line utility to stop the VM. As another example, the virtual UEFI boot sequence doesn’t act as some operating systems expect, which can result in the user getting dumped into the UEFI shell. (Something normal users should never see!)

UEFI Shell

Google pointed me to this page which will help with most Linux distributions that encounter this problem. Thanks to this tip, I got my Ubuntu test server up and running on a FreeNAS VM.

Upside: Full virtual machine isolation will protect against most security exploits.

Downside: Full virtual machine isolation consumes more system resources. Most significantly, the storage is not shared: Space dedicated to a VM is not available to the rest of FreeNAS. And there is no storage mapping so a VM could only access FreeNAS shares over the network interface as if it is physically a different machine.

Summary: Virtual machines make sense for things that do not interact with the rest of FreeNAS, and a good alternative to setting up another physical machine.

FreeNAS Plugin: Plex Media Server

plex-logo-e1446990678679Once I had a few simple network file shares set up in FreeNAS, it was enough to do most of what I want in a home network storage device. For a fraction of the cost of a commercial solution like Drobo. Now we can start looking at the less critical fun stuff.

Part of my home media collection stored on my NAS includes various video files that I’ve been carrying around. Most of them were standard-definition video files I recorded off of broadcast television programs. This was done at a time when most people would record to VHS tape. Only the super nerdy types record to computer video.

So I had the files, but I didn’t have a good way to play them straight off a file share. This is where something like Plex comes in. There’s a server-side component that runs on my FreeNAS box, talking to client-side components for various devices. The web client could cast to a Google Chromecast, and the Amazon Fire TV stick has a Plex client app.

For security isolation, FreeNAS runs plugins inside a “jail”. This is a FreeBSD feature that sounds a lot like a Docker container but isn’t a Docker container. This isolation is good default security, but it does mean the Plex Media Server plugin could not see the rest of the FreeNAS box until the user specifies a way for the code inside the jail to see specifically allowed files outside of a jail. I could even specify the storage visibility to be read-only so there would be no accidental manipulation of my video files.

Once I got past the FreeBSD jail mechanics, it was mostly smooth sailing. The only problem came from the large fraction of my files encoded in Windows Media Video format, an old video format that Plex does not support. If I end up deciding I like the Plex experience, I will have to look into doing a bulk re-encode of these old video files.

FreeNAS File Sharing: Trust the Wizard

FreeNAS LogoThe authors of FreeNAS tries to make things easy for the user by providing automation tools (“Wizards”) that take care of the fine administrative details without requiring the user to learn all the underlying nuts and bolts of FreeNAS, or FreeBSD, or Linux kernel, etc.

This is especially true for creating network file shares in FreeNAS. It supports many network file sharing protocols. Including the Apple-specific AFP (Apple File Protocol), the Microsoft Windows-based SMB (Server Message Block), the Unix-based NFS (Network File System), plus three others I don’t even understand.

Each of these have their own setup requirements that a casual user like myself is unlikely to get right on our own. So the manual encourages the use of Wizards. (Bold emphasis mine, but I think it should be in the manual!)

FreeNAS® provides a Wizard for creating shares. The Wizard automatically creates the correct type of dataset and permissions for the type of share, sets the default permissions for the share type, and starts the service needed by the share. It is recommended to use the Wizard to create shares.

Windows has a file sharing wizard as well: In Windows file explorer, I would create the folder I want to share, then right-click on that folder to select “Sharing…” This launches the wizard who will then take care of everything else.

Since I was used to the above workflow, I did the same thing in FreeNAS. Create a directory, then run the wizard to share that directory. Unfortunately this results in network shares that were not accessible. (“Access Denied”)

I eventually debugged the problem to my “create a directory” step. Since I had created as an administrator, the permissions on that directory were not set for use by other users. And the FreeNAS wizard did not (or could not) update the permissions properly.

What I needed to do was to launch the FreeNAS network sharing wizard, and tell it to create the directory as part of the network share creation process. This way the directory would be created by the wizard who will properly set the permissions for file sharing.

I did too much and that became unhelpful.

Trust the Wizard.

FreeNAS USB Flash Boot Drives: Recovering Boot Drives That Don’t Boot.

FreeNAS LogoWe’ve established that FreeNAS can mirror the USB flash drive boot device for redundancy. If one of them should fail, the system can be recovered with the other. I hadn’t set out to verify the recovery procedure, but I stumbled into a practice run anyway. In hindsight it’s good to get a feel of my recovery options now when I’m still playing with the system, rather than later when I’m in a panic to recover my data.

Over the past few days I’ve experimented with the boot volume, setting up the mirroring and using the built-in replacement mechanism to retire my USB sticks with checksum errors. After this was done, I unplugged the checksum-error drives (including the one I originally installed FreeNAS on) and rebooted the system.

It failed to boot and ended up at the GRUB recovery menu. Hmm, that’s not good.

I plugged all the old drives back in… and that didn’t improve the situation. Apparently, in the midst of all the mirroring and replacing, I managed to damage GRUB configuration so FreeNAS no longer boots.

Since I expected all the actual operating system files to be OK, I searched for a way to rebuild just the boot loader portion of the USB sticks. There are many ways to do this. The easiest – looking route I took is to perform an in-place upgrade.

I booted up the FreeNAS installation media, and selected the “Install/Upgrade” option. I pointed it to a USB drive that was free of checksum errors but would not boot. The installer detected an existing installation and offered to generate a new boot environment for the existing FreeNAS installation. Sounds perfect! I hit <OK>

Ten minutes later, the upgrade failed due to an error writing boot configuration files the installer wants to add. A quick Google search found a few suggestions on how to fix it using command-line tools I wasn’t familiar with. I decided to try easier things first.

I restarted the installation process and chose the other upgrade option: An upgrade with a disk reformat rather than just generating a new boot environment. The installer will preserve a copy of the FreeNAS configuration but wipe everything else from scratch.

This approached was more destructive and took more time but it worked. My FreeNAS box booted back up as if nothing happened. Success!

FreeNAS USB Flash Boot Drives: Wide Variations in Performance

Since I went into this FreeNAS project with the expectation to experiment and learn, I followed the recommendation to use commodity USB flash drives as the operating system boot drive despite my skepticism. The previous blog post discussed checksum errors found on some of the USB flash drives I had on hand, and how FreeNAS is able to mitigate the errors by mirroring the operating system across multiple USB sticks. This greatly increased my confidence entrusting the FreeNAS operating system to these USB thumb drives.

Usually these devices are used for ferrying files from one computer to another. They expect to see some files copied onto the drive, then copied off the drive in order. (sequential read/write operations.) Putting them in service as the operating system drive is an entirely different work pattern, with small pieces of data written at unpredictable places and other data retrieved from equally unpredictable places. (random read/write operations.)

I was aware of this difference and it was part of my skepticism using USB sticks as operating system drive. Now that I’m using it, I can see how it works in reality. Thanks to FreeNAS boot device mirroring, I have the confidence to go into this experiment without risking my data.

Outside of errors or outright failures, the other thing I had expected was degraded performance. If the flash memory controller chip is optimized for sequential operations, it might be bad at random operations. Unfortunately there was no way to tell up front. The software running on the flash drive controller isn’t something manufacturers put on the packaging. And every company has their own algorithm.

Since they are so cheap, the easiest thing to do is to just go and try it. The result is clear: some USB flash memory drive controllers are far far better at this workload than others.

Some of this difference is felt immediately, in the time taken for the mirroring duplication process. During this “resilvering” process roughly 2GB of data is copied to the new flash drive. Out of the two drives that had no checksum errors, one of them was able to complete the process in a little over two hours. The other took 26 hours, more than ten times longer!

Their difference is also visible in the FreeNAS system performance reporting page. Since FreeNAS keeps the two drives in sync, it’s easy to see how they respond to identical workloads. The flash drive identified as (da1) breezes through work, never more than 50% busy. Its mirrored sibling (da2) struggles to complete the same work, frequently spiking up to 100% busy.

DiskBusy

One of these is much less happy at their new job than the other.

Between the checksum errors and the disk busy graph, I am now much better enlightened on the varying quality of USB flash drives. I had thought of them as all basically equivalent, so I just bought whichever is the cheapest. My attitude has now changed. From here on out, whenever I need to buy USB flash drives, I’ll look for those made by SanDisk.

FreeNAS USB Flash Boot Drives: Mirroring For Fault Tolerance.

FreeNAS LogoFreeNAS encourages the use of USB flash drives as the operating system boot drive. This allows FreeNAS to dedicate all of the motherboard SATA connectors for data storage drives. I didn’t think commodity USB flash drives are trustworthy enough to hold the operating system, but I was willing to experiment and be proven wrong.

The very first night, I got worrying news from the nightly system check:

  pool: freenas-boot
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 3K in 0h10m with 0 errors
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0    11

errors: No known data errors

Looking on the bright side, “No known data errors” is comforting, as is the “repaired […] with 0 errors”. It’s nice FreeNAS was able to repair whatever was wrong with my USB stick. I suspect inexpensive commodity USB flash drives frequently encounter errors that are silently corrected by the operating system. Still, an error is an error and it’ll only be a matter of time before I run into a serious problem.

Fortunately, FreeNAS authors had the foresight to make sure a bad boot device does not become a single point of failure. A second one can be added to the system act as a mirror to the boot device. If either of them fails, the other can take over.

Much to my dismay, the second USB stick I tried also encountered a data checksum error. I didn’t have much luck figuring out how to interpret the checksum error code, but I did learn that it is supposed to be zero. The first stick returned 21, the second 26.

I tried a third USB stick and was relieved to finally see a zero checksum. The output below was generated when I ran ‘zpool status’ while the third stick is in the middle of replacing the second stick.

  pool: freenas-boot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress
config:

        NAME             STATE     READ WRITE CKSUM
        freenas-boot     ONLINE       0     0     0
          mirror-0       ONLINE       0     0    21
            da0p2        ONLINE       0     0    21
            replacing-1  ONLINE       0     0     0
              da1p2      ONLINE       0     0    26
              da2p2      ONLINE       0     0     0  (resilvering)

errors: No known data errors

I also found a fourth USB stick that was checksum error-free, so I had it take the place of the first one.

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h29m with 0 errors
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0    21
            da1p2   ONLINE       0     0     0
            da2p2   ONLINE       0     0     0

errors: No known data errors        

Now both boot drives in the mirror set have zero checksum error, but the mirror volume overall still has checksum error 21 from the first USB stick. I’m still learning if that means anything (bad) and what it would take to reset that to zero.

FreeNAS Box v2: Component Access

One aspect that was completely neglected in v1 was any kind of an access door. The panels were designed only enough for them to mesh, so the only way to hold things together is to glue everything shut. Completely impractical! The only way to perform any maintenance would be to shatter some acrylic to break the case open.

Now we have two access panels, one in the front and one on the bottom. I tried to see if I could somehow integrate things so there would only need to be one access panel, but never came up with a design that would work well while simultaneously satisfying all the other design goals I’m trying to accomplish.

Each access panel is held in by 4 x #6-32 screws, the standard desktop PC case screw. They fasten into threads I had tapped into the underlying layer of plastic. I should use heat-set inserts for better durability. Unfortunately I didn’t have #6-32 inserts on hand at the time and I thought I would probably build a V3 follow-up anyway.

FreeNAS v2 Front Panel Open

The front panel opens to allow access to the front chamber where the motherboard and CPU lives. If I wanted to upgrade the memory or switch out the SATA cables, I could do so through the front panel opening.

The bottom panel opens to allow access to the rear chamber. The two hard drives and the power supply are all installed through the bottom opening. The drives can be individually replaced without fighting through too many pieces of cable. The power supply can also be replaced through the opening exposed by the bottom panel.

FreeNASv2 Both Panel Open

FreeNAS Box v1 to v2 Size Comparison

Now that FreeNAS Box v2 is up and running, let’s do a size comparison to see how things have changed. The width dimension was a regression: v2 is wider by 3 centimeters. The real space requirement increased even more than that, because the v2 air intakes are on the sides.  So it needs additional room to the left and right of it in order to avoid blocking those intakes. In contrast, v1 with its bottom intake would be OK sitting flush against objects to its left and right.

Looking at the final results in reality, I think I can rearrange a few things to reduce the width by 1 cm. Something to consider for a potential FreeNAS Box v3.

FreeNAS v1v2 SideBySide

The greatest improvement is in the height which has been reduced by 7 cm thanks to the elimination of the lower air intake cavity. The advantage is somewhat reduced when in use, because the power cable now sticks straight up and adds a bit to the required height. If this becomes a serious problem, though, I could always switch to an power cable with a right-angle plug. This will allow the box to fit in a shelf only 28 cm high, leaving barely enough for proper heat exhaust. Looking over the results, I think I can safely reduce the bottom of the case by 1 cm and everything will still fit, another change for the potential v3.

FreeNAS v1v2 SideBySide

And finally, the depth is reduced by 3.5 cm. Real world improvement is even more, because now the back of the box can sit flush against the wall in a way that v1 could not.

The reduced height and depth has more than compensated for the increase in width, v2 is overall a more compact space-efficient package than the v1 design.

FreeNAS Box v2: Construction Complete

After spending an afternoon + evening at a Tux-Lab work session, I have my FreeNAS Box v2! It’s always fun to see my idea turned into reality.

FreeNASv2 Complete

The first thing I appreciated was the fact that the components are clearly visible through the acrylic panels. And even better, the messy tangle of wires are hidden behind them. This reversal from v1 is the best aesthetic change.

The other major design requirement – that both cooling fans be visibly spinning – is also present but it doesn’t have as much of an immediate aesthetic effect.

After I’ve kept it on and running overnight, I checked the temperature around the box the following morning. I think it would be neat to check thermal performance with a FLIR thermal imaging camera but lacking such toys I went with the low-tech way of putting my hands at various places around the box to feel the temperature.

The front chamber – where the CPU and motherboard reside – has a slight temperature gradient from top to bottom but overall it was relatively cool to the touch. This was expected as the CPU basically sat idle all night. It also means I won’t need to cut a hole in the front door for a direct air intake.

The rear chamber, with the power supply and both hard drives, is where most of the heat is generated. The two drives were warm to the touch signifying that they’ve been spinning all night and getting some amount of cooling to keep them from getting hot.

The power supply fan was running and the power supply case was cool to the touch. The power meter read 30W for the FreeNAS box in this steady-state idle state. This is a very light load for a 600W-rated power supply, reflected in its cool running temperature.

FreeNAS Box v2: Construction Fixture

One of the problems with FreeNAS Box v1 was that I designed it with tabs and slots to fit into each other. While it made the box easy to assemble, the slots severely weakened the structure of the box.

For FreeNAS Box v2 I avoid the tabs and slots. But I need something else in their absence to help me during construction. The answer is a fixture: Something I design along with the box that helps me build it, but not part of the end product.

Building the box will start with bonding all the major vertical pieces together. Once the cement has set hard enough for them to stand alone, the fixture pieces can be removed. The resulting assembly will then be self-supporting as the remaining pieces are attached.

The fixture pieces sit top and bottom. Pretty much where the largest horizontal pieces would eventually go, but are distinctly different from those pieces.

FreeNASv2 Fixtures.JPG
Initial assembly (gray) with assembly fixture (yellow)

The top fixture has two slots for holding two of the vertical sheets of acrylic. We’ve already established such slots are bad for the structural strength of the end product, but it’s perfectly OK (and quite useful) to have them in a fixture.

Both the top and bottom fixture have round cutouts in the corners and in the mid-span T-joint so that they stay clear of any extraneous acrylic cement that might leak out. This way we avoid accidentally cementing the fixture to the product.

Each of the fixture is made of two layers of acrylic, a main layer and a secondary layer whose shapes helps keep the box pieces in place. The small round circles visible in the picture is sized for M4 screws to fasten the fixture layers together. Using screws instead of acrylic cement allows us to later disassemble the fixture and recycle the pieces as scrap acrylic in future projects.