Switching to ESP32 For Next Exercise

After deciding to move data processing off of the microcontroller, it would make sense to repeat my exercise with an even cheaper microcontroller. But there aren’t a lot of WiFi-capable microcontrollers cheaper than an ESP8266. So I looked at the associated decision to communicate via MQTT instead, because removing requirement for an InfluxDB client library meant opening up potential for other development platforms.

I thought it’d be interesting to step up to ESP8266’s big brother, the ESP32. I could still develop with the Arduino platform with an ESP32 but for the sake of practice I’m switching to Espressif’s ESP-IDF platform. There isn’t an InfluxDB client library for ESP-IDF, but it does have a MQTT library.

ESP32 has more than one ADC channel, and they are more flexible than the single channel on board the ESP8266. However, that is not a motivate at the moment as I don’t have an immediate use for that advantage. I thought it might be interesting to measure current as well as voltage. Unfortunately given how noisy my amateur circuits have proven to be, I doubt I could build a circuit that can pick up the tiny voltage drop across a shunt resistor. Best to delegate that to a dedicated module designed by people who know what they are doing.

One reason I wanted to use an ESP32 is actually the development board. My Wemos D1 Mini clone board used a voltage regulator I could not identify, so I powered it with a separate MP1584EN buck converter module. In contrast, the ESP32 board I have on hand has a regulator clearly marked as an AMS1117. The datasheet for AMS1117 indicated a maximum input voltage of 15V. Since I’m powering my voltage monitor with a lead-acid battery array that has a maximum voltage of 14.4V, in theory I could connect it directly to the voltage input pin on this module.

In practice, connecting ~13V to this dev board gave me an audible pop, a visible spark, and a little cloud of smoke. Uh-oh. I disconnected power and took a closer look. The regulator now has a small crack in its case, surrounded by shiny plastic that had briefly turned liquid and re-solidified. I guess this particular regulator is not genuine AMS1117. It probably works fine converting 5V to 3.3V, but it definitely did not handle a maximum of 15V like real AMS1117 chips are expected to do.

Fortunately, ESP32 development boards are cheap, counterfeit regulators and all. So I chalked this up to lesson learned and pulled another board out of my stockpile. This time voltage regulation is handled by an external MP1584EN buck converter. I still want to play with an ESP32 for its digital output pins.

Move Calculation Off Microcontroller to Node-RED

After I added MQTT for distributing data, I wanted to change how calculations were done. Using a microcontroller to read a voltage requires doing math somewhere along the line. The ADC (analog-to-digital) peripheral on a microcontroller will return an integer value suitable for hardware registers, and we have to convert that to a floating-point voltage value that makes sense to me. In my first draft ESP8266 voltage measurement node, getting this conversion right was an iterative process:

  • Take an ADC reading and convert to voltage using a conversion multiplier.
  • Comparing against voltage reading from my multimeter.
  • Calculate a better conversion factor.
  • Reflash ESP8266 with Arduino sketch that includes the new conversion factor.
  • Repeat.

The ESP8266 ADC is pretty noisy, with probable contributions from other things like temperature variations. So there was no single right conversion factor value, it varies through time. The best I can hope for is a pretty-close average tradeoff. While looking for that value, the loop of recalculating and uploading got to be pretty repetitious. I want to move that conversion work off of the Arduino so it can be more easily refined and updated.

One option is to move that work to the data consumption side. This means logging raw ADC values into InfluxDB and whoever queries that data is responsible for conversion. This preserves original unmodified measurement data allowing the consumers to be smart about dealing with jitter and such. I like that idea but not ready to dive into that sort of data analysis just yet.

To address both of these points, I pulled Node-RED into the mix. I’ve played with this flow computing tool earlier and I think my current project aligns well with the strengths of Node-RED. The voltage conversion process, specifically, is a type of data transformation people do so often in Node-RED that there is a standard node Range for this purpose. Performing voltage conversion in a Range node means I could fine-tune the conversion and update by clicking “Deploy” which is much less cumbersome than recompiling and uploading an Arduino sketch.

Node-RED also allows me to carry both the original and converted data through the flow. I use a Change node to save original ADC value to another property before using Range to convert ADC value to voltage. Now I have a Node-RED message with both original and converted data. Now I need to put that into the database, and I searched the public Node-RED library for “InfluxDB” and I decided to try node-red-contrib-stackhero-influxdb-v2 first since it explicitly supported version 2 of InfluxDB. I’m storing the raw ADC values now even though I’m not doing anything with it yet. The idea is to keep track so in the future I can explore voltage conversion on the data consumption side.

To test this new infrastructure design using MQTT and Node-RED, I’ll pull an ESP32 development board out of my pile of parts.

Here is my Node-RED function to package data in the format expected by node-red-contrib-stackhero-influxdb-v2 InfluxDB write node. Input data: msg.raw_ADC is the original ADC value, and msg.payload the voltage value converted by Range node:

var fields = {V: msg.payload, ADC: msg.raw_ADC};
var tags = {source: 'batt_monitor_02',location: 'lead_acid'};
var point = {measurement: 'voltage',
      tags: tags,
      fields: fields};
var root = {data: [point]};
msg.payload = root;
return msg;

My simple docker-compose.yml for running Node-RED:

version: "3.8"

    image: nodered/node-red:latest
    restart: unless-stopped
      - 1880:1880
      - ./data:/data

Raspberry Pi GPIO on Node-RED Needs Raspberry Pi OS

I didn’t manage to get Bluetooth Low Energy (BLE) running in Node-RED on a Windows PC, so experimentation moved on to other hardware platforms. Specifically the hardware hacking darling Raspberry Pi. I knew it was possible because of the number of projects that already existed, plus Node-RED has an instruction page dedicated to Raspberry Pi installation. This would be straightforward… except I intentionally made things difficult for myself.

The twist is that my Raspberry Pi 3 readily available for experimentation was most recently set up to check out ROS on arm64. Which meant it was running Ubuntu 20 for 64-bit Raspberry Pi instead of the default Raspberry Pi OS. (Formerly known as Raspbian.) I was confident all the fundamental parts of Node-RED such as flows and network I/O would continue functioning, which makes it a low-power alternative to running Node-RED on a PC. But the other reason to run Node-RED is to interface with devices via Raspberry Pi’s GPIO pins. I wanted to know how well that would work under arm64 Ubuntu.

And I got my answer: it does not work. It appears whatever means Node-RED employed to access Raspberry Pi GPIO is not available on Ubuntu arm64 builds. I assume it is possible to install and/or port over the components required for such support, but today’s experiment is just to see if it works out of the box and now I know it does not.

31 Aug 20:36:53 - [info] Node-RED version: v1.1.3
31 Aug 20:36:53 - [info] Node.js version: v12.18.3
31 Aug 20:36:53 - [info] Linux 5.4.0-1016-raspi arm64 LE
31 Aug 20:36:54 - [info] Loading palette nodes
/bin/sh: 1: /home/ubuntu/.node-red/node_modules/node-red-node-pi-gpio/testgpio.py: not found
31 Aug 20:36:56 - [warn] rpi-gpio : Raspberry Pi specific node set inactive

Adding Noble Specified Hardware Failed To Enable BLE Discovery

I’ve known Bluetooth Low Energy (BLE) to be a new technology that can still be challenging to interface with. I’ve had success so far with Node-RED making hard things easy, most recently in reading battery power state in an old Windows 10 laptop. So I wanted to see if BLE can be just as easy. The answer: it was not. Getting node-red-contrib-noble-bluetooth set up and launching a flow was easy, but attempting to discover nearby BLE devices caused Node-RED itself to crash.

No compatible USB Bluetooth 4.0 device found!

Looking at the crash stack, the culprit appears to be Noble, the Node.js BLE module upon which the Node-RED extension was built. There are some native code components that were built for Noble support, and if things go sour in native code, it fails like native code, hence the jarring failure. Quite unlike the typical recoverable JavaScript exception I’ve become spoiled by.

Searching on that error message, I found more information in this GitHub issue filed against Noble, which pointed to a Noble document explaining that support is limited to a fixed list of hardware. Not surprisingly, the chip on board my experimental laptop was not on the list. I searched for the hardware on that list, and thought the Asus USB-BT400 (*) was cheap enough to order and give it a try.

For the BT400 to function under Noble, I also needed to circumvent the normal driver installation process and instead install WinUSB so Noble can directly access the hardware to bypass Windows’ Bluetooth stack. The WinUSB installation link in Noble Wiki is dead, but a web search pointed to this link as the modern replacement.

Once WinUSB driver was installed for the Asus USB-BT400 Bluetooth adapter, Noble was able to find it and didn’t crash when I attempted to discover nearby BLE devices. Unfortunately, neither did it find any of the BLE hardware I had on hand for this test. I had hoped that, even if I couldn’t use that hardware, I could at least discover their presence.

This test was not a success, ah well. Life moves on to other experiments.

(*) Disclosure: As an Amazon Associate I earn from qualifying purchases.

Node-RED Challenge Round 2: Bluetooth Low Energy

My first challenge to Node-RED was a success: I was able to read battery voltage and charge percentage from a slow Samsung 500T tablet stuck on an old 32-bit build of Windows 10, and the hardest part of that process was retrying installation after it timed out because the computer was so slow.

This next challenge is significantly more difficult: connect to peripherals via Bluetooth Low Energy. Despite Bluetooth in the name, BLE is actually a completely different protocol from the earlier grand wireless protocol to rule them all. (Sometimes called “Bluetooth Classic” now.) But it is now administered by the same consortium,. so there we go.

As its name implies, a primary goal for BLE is reducing power requirements to make it feasible for battery powered devices. And in this context “battery” is not a gigantic brick of rechargeable lithium-ion cells, BLE wants to be practical for devices to run for months on little coin cell batteries. It’s new, with its own set of rules, and tricky to get right. Thus the perfect advanced level challenge.

This time the hardware is the HP Split X2 from NUCC, an old Windows laptop with built-in Bluetooth. It has a decent processor and RAM but hobbled by an old hard drive that’s difficult to upgrade. As a result installing Node-RED took almost as long as it did on the Samsung 500T’s slow eMMC storage, but at least the CPU was fast enough to avoid a timeout.

The Node-RED extension of interest here is node-red-contrib-noble-bluetooth. Out of all the nodes claiming Bluetooth capability, this one seems to be the one that has general BLE capability (not tied to specific devices) and updated most recently. I installed that extension, started a query for nearby BLE devices, and Node-RED crashed. Not throwing an error that a flow might try to handle, not an error message, Node-RED itself crashed with an error thrown by Noble.

No compatible USB Bluetooth 4.0 device found!

Ah well, I didn’t expect it to be as easy as reading battery level and fully anticipated some bumps on the road to this advanced challenge. Time to do a little investigation.

Approaches to Optimizing Samsung 500T For Solar Power

My Samsung 500T solar power project is an exercise to maximize a battery’s ability to help store and use solar power by creating a usage regimen maximizing its cycle life. I’m doing this on the 500T because it was available and I wouldn’t cry too much if I make a mistake ruining it. Hopefully ruination is unlikely, as I plan to leave the built-in battery management circuit intact to save me from drastic mistakes.

According to Battery University, best longevity of lithium batteries comes when they avoid the stress of either end of their capacity. This means avoiding charging to full, and avoiding discharging to empty. A common approach is to charge until 80% and no further, leaving 80% – 100% unused. And when discharging, go no lower than 20%, leaving 20% – 0% unused. Using just the “middle” 60% means the battery would have to go through almost twice as many charge/discharge cycles to deliver the same overall energy, but doing so is rewarded with a battery that can survive much more than twice the number of cycles so we come out ahead.

Maintaining such a protective regimen is a hassle that isn’t usually worth the trouble for consumer electronics, because they’re usually replaced within a few years for reasons other than battery degradation. But such maintenance is done for batteries intended for longer lifespan, like those on satellites and electric cars. My 2012 Chevy Volt does this in a way transparent to the driver: When my car told me the battery is empty, it’s actually somewhere around 20% and when it told me the battery is full, it’s actually roughly 80%. The Samsung 500T does not do this, which we could verify by watching both the proclaimed charge percentage and its actual voltage.

Handling the low end should be easy – watch for the charge percentage to drop to 20% and connect power to lead acid array for charging. The high end is a little trickier. Ideally I could attach power to the 500T and charge it slowly up to 80% during daylight hours, but I don’t have that kind of control. When I connect power, it will charge to 100% as fast as it could. I could only cut power at 80% charge to avoid going into the upper 20%. This means we’ll end up with an extra discharge cycle during the day, but if the overall power regimen actually improves cycle life by more than triple, it will still be a worthwhile trade off.

Constraints on Optimizing Samsung 500T For Solar Power

I monitored a battery discharge and recharge cycle on a Samsung 500T to get a feel on how it can be a better partner in my mini solar system. Right now it is constantly plugged in to the lead-acid battery array, leaving its own internal battery constantly full. I want to make use of that battery capacity to help store solar power for the 500T’s current task of displaying ESA’s space station tracker.

For this first draft I want to avoid cracking the computer open, which means I don’t have any control over the charging algorithm. It will charge at full speed whenever I connect power, and it will try to charge to 100%. Charging to full is stressful for lithium ion batteries and decreases their lifespan. Some computers offer the ability to halt charging at 80% reducing use of capacity in exchange for extending the usable life of the battery. Unfortunately I saw no such option with the 500T.

The discharge rate is also pretty much fixed, as I’ve turned off all I can of Windows and Samsung crapware. Leaving just the ISS tracker and Node-RED running. About the only parameter I can adjust is the rate at which Node-RED queried data, and I think I can back off from once per minute to maybe once per three or five minutes. This will help a tiny bit.

The good news is that, watching through the data retrieved by Node-RED through a discharge and charge cycle, I see nothing worrisome about that data and willing to trust it. With data in hand, there are two more things to bring in to the picture: (1) a switch for the tablet’s power cord to my lead acid batteries, and (2) a program to connect/disconnect that power at the best times to optimize the system. I want to think over the latter before I spend money on the former.

Monitoring Samsung 500T Discharge-Charge Cycle

Getting through Node-RED installation on my Samsung 500T tablet was the hard part. Once done, it was trivial to set up a flow to extract the tablet’s battery voltage and charge percentage. I didn’t want to add any more overhead than I already have, so the flow sends that data off to a MQTT broker. Allowing me to analyze that data on a different computer without impacting the battery consumption on the 500T.

It was instructive to watch those two graphs after I unplugged the 500T, charting its battery discharge down to under 10% followed by a charge back up to 100%. During this time, the 500T kept its screen on displaying ESA’s ISS Tracker, plus the normal Windows background tasks. And now it also has Node.JS running Node-RED to query the battery voltage and charge percentage.

The first observation is that the battery discharge percentage graph is impressively linear. As a user I never felt this way intuitively, as my most memorable episodes are battery meters whose value seem to drop faster and faster as I raced to finish a task before my battery gave out. A linear graph is impressive because a lithium ion battery’s discharge voltage is not linear, something we can see on the voltage graph. It drops sharply off 8.4V before stabilizing on a gentle slope that’s more like a curve as it gradually slowed approaching 7.4V. (That is the commonly listed nominal voltage level for two lithium ion battery cells in series.) Once it dips below 7.4V, voltage curve starts dropping rapidly and trending towards a steep dive when I plugged the 500T back in to a power source. We can also see that the voltage level is a bit noisy, fluctuating as it discharges. In contrast, except for a little dip off 100%, the percentage graph is steadily and reliably decreasing under a constant workload. Just as we’d expect, with no surprises. I have a lot of complaints about this machine, but its power management is rock solid.

For the charge cycle, again the percentage value is not based solely on battery voltage as we see those two are wildly divergent during charging. When I plugged in to recharge, there was a big jump upwards as the machine switched to charging cycle. Towards the end of that cycle as charge state approached 100%, there was some kind of a top-off procedure. I was surprised to see that charge controller allowing the battery voltage to exceed 8.4V, something I’ve been taught to never do when charging bare 2S LiPo battery packs. But according to this voltage graph, exceeding that voltage was part of the process of filling the battery before letting it settle down to around 8.35V. All through this top-off procedure, the battery percentage reported 100%.

I enjoyed this opportunity to geek out over battery management minutiae, but it isn’t just for curiosity’s sake. There was a project idea behind all this.

Installing Node-RED on Samsung 500T

When I saw that Node-RED community contributions included a general battery state-of-charge reporting node, I thought that would be a practical and useful first step into playing with contribution nodes. I wanted to give that specific ndoe a try because given my past miserable user experience on my Samsung 500T tablet, I was not terribly motivated to put in the effort required to write my own code to extract battery information. But I’m willing to see if that Node-RED node can work its magic here, and the first step of is, obviously, see if we can even install Node-RED on the unloved machine.

There were many reasons to be pessimistic. Windows was and is still seen as a second-class citizen in many web-centric products. And not only is the tablet running Windows, it is stuck on an old version of Windows because of the whole Intel Clover Trail mess preventing modern Windows 10 support. Furthermore, Clover Trail is a 32-bit only chip without support for x86-64 a.k.a. amd64 instructions that are starting to become a requirement on modern software frameworks. And it only has 1GB of RAM. And we’re still dealing with that molasses-slow eMMC storage. The list goes on.

Fortunately Node.JS still has a 32-bit Windows installer, though installation failed after nearly an hour of grinding away. I was first disappointed but then relieved to see it was “merely” a timeout installing one of the auxiliary components. This was not a huge surprise given how slow this machine is, so I waited for the system to settle down (lots of backlog waiting on that eMMC) before retrying. Node.JS installation succeeded the second time and after that I installed Node-RED without drama.

Launching Node-RED faced another unnecessary hurdle in the form of Windows Firewall. Annoyingly, this was one of the differences between Windows 1607 and modern builds, so I had none of the menu options I’ve become familiar with. But eventually I was able to let Node.JS open up a port to serve the Node-RED user interface, where I could install node-red-contrib-battery.

I dropped three nodes into the blank flow: an inject node, a debug node, and the newly installed battery node between them. I hit “Deploy” and clicked to see if that battery node can extract any information and… it can! Not a lot, as most of the fields are blank or null, but it did return the current battery voltage and the charge percentage. This is all I need to observe this tablet’s charge and discharge behavior.

Node-RED Challenge Round 1: Battery Level Reporting

When I discovered Node-RED and its vast library of community contributions, I checked for several pieces of functionality that might enable several different projects ideas on my to-do list. One of them was the ability to query battery discharge level on a computer.

This was a follow-up to my earlier project assigning a Samsung 500T tablet to display an ISS track around the clock. When I last left that project, I modified the tablet so it runs on my small solar power array. But the arrangement was not optimal: the tablet is constantly pulling power from the array storage battery, instead of contributing its own battery to help store solar energy.

I had the thought to integrate that battery in an automated fashion. There are several ways it could go, but an important data point is battery discharge level. It is the key piece of information in any algorithm’s decision to charge the tablet during the day and let it run on battery at night.

This is something I knew I can accomplish by writing a bit of code. I need to find the APIs to query battery level, then I need to write code to extract that information and do something with it. None of that were technically challenging tasks, but I never allocated the time to do it.

Now I have Node-RED and its library of tools, which included node-red-contrib-battery purported to report battery information in a common way across Linux, MacOS, and Windows. If it works as advertised, all that battery query coding I kept putting off could be as simple as hooking up some nodes. It’s definitely worth trying.

Node-RED Community Contributions

Evaluated in isolation as a novel way to program computers, Node-RED scores high. I got up and running with my serial communication projects much more quickly than writing code in a UWP application, it was easy to plot a graph of data fed by my Mitutoyo SPC connector project. While I did have to climb the learning curve of a new way of thinking and a new set of vocabulary, but once I climbed it (pretty shallow, all things considered) I understood another huge attraction of Node-RED: the collection of community contributions.

I got a brief glimpse of this when I installed the Node-RED Dashboard extension, where I went in to the extension menus to search for Dashboard. The fact there was a search function implies a sizable number of extensions, so I made a note to check it out later. This was affirmed when I went to search for the serial port node, but again I put it off to later.

Returning to browse the “Flows” directory of Node-RED, I’m very excited by the extensive toolbox people have shared and made easily usable by others. This is a clear sign of a virtuous cycle at work: an attractive and useful tool attracts a seed group of users on its own merits. These users share their improvements, making the tool more useful and attractive to other users, and the cycle repeats until we have a big toolbox with contribution by people everywhere.

I queried for functionalities that I knew I would need for many projects on the hypothetical to-do list. Majority of queries came back with something that looked promising. After a few successful hits in a row, I was half expecting to find a Node-RED extension to control a webcam attached to a 3D printer carriage. Alas, no such extension existed to trivialize my current project. Fortunately, there’s a community contributed battery information node that could pick up where a past project left off, so I’ll try that first.

Tracking History of a Node-RED Project

Exploring Node-RED has been a lot of fun, working with both the freedoms and limitations of a different way of programming. It was novel enough that a very important part of software development workflow didn’t occur me until I had a nontrivial program: Without source code text files, how do I handle source control? Thankfully it turns out there are still text files behind the scenes of graphical flows, and I can use them with git just like any other software development work.

There are two options: doing it automatically via integrated git client, or manually. The procedure to setup integrated git client is outline in Node-RED documentation but I’m not sure I’m ready to trust that just yet. There’s an additional twist: it requires that I have my git credentials sitting on the computer running Node-RED, which isn’t something I’m necessarily willing to do. I keep a strict watch over the small set of computers authorized with my git credentials for software work. In contrast, a computer running Node-RED is likely only running Node-RED and not used for any other development. (This is especially true when Node-RED is only running in a FreeBSD Jail on my home FreeNAS server, or a Raspberry Pi.)

As a side note, the git integration instructions did explicitly confirm something I had suspected but hadn’t found confirmation before: “Node-RED only runs one project at any time.” This makes the FreeBSD Jail approach even more interesting, because running multiple isolated Node-RED projects on a single physical machine can be done by keeping each in their own jails.

Back to source control: for the immediate future, I’ll use a manual workflow. Under Node-RED’s main menu is the “Export” option. I can click “all flows” to include every tab, click “formatted” to make it readable, and click “Download” to receive a text file (in JSON, naturally) representing my Node-RED project.

By doing this on one of my machines configured for git access, I can put this file in a git repository for commit and push operations. Regretfully, the file is not terribly readable in its native form, and the GitHub text diff mode is cluttered with a lot of noise. There are many generated id fields linking things together, and those ID tend to change from one download to the next. However, it is far better than nothing and at least all the important changes are also visible within the noisy churn.

To verify that I could restore a project, I set up Node-RED on another computer and imported the flows. All the nodes in my visible flow appear to have survived the transition, but I’ve run into some sort of problem with the configuration nodes. Serial communication nodes lost their COM port information like baud rate, timeout, and line termination. This is odd, as I could see that information in the exported JSON. Similarly, my dashboard layout has been lost in the transition. Hopefully this is only a matter of a beginner’s mistake somewhere. For now it is relatively easy to manually restore that information, but this would quickly become a big headache as a project grows in size.

[I have no idea why anyone would want it, but if someone desires my air bubble packing material squish test flow, it is publicly available on GitHub.]

Packing Bubble Squish Test Data

I didn’t expect much out of a silly “Hello World” test of a machine that squishes packing material, but I underestimated how much of a geek I am for data. Raw numbers out of the load cell didn’t mean much, partly because it was so noisy. But since it was trivial to send raw HX711 readings to a Node-RED chart for visualization, I plotted load cell pressure data over time and was surprised at what I could see in that graph!

The most obvious thing is that we can definitely see each downward stroke of the machine represented as a sharp downward spike in the graph. After that initial shock, though, the air bubble started to relax and we can see a reduction in pressure transferred to the plate. This is a trend that I couldn’t see just looking at raw numbers flying by, and a good visual (numerical?) representation of what happens with “items may have settled in shipping”.

What I did not expect ahead of time, but was pretty obvious in hindsight, is the visible trend from one stroke to the next. The bubble bounced back incompletely when the machine released. Therefore each stroke resulted in a lower transmitted force than the last, with a degradation curve across multiple strokes that echoes the pressure reduction visible within each stroke.

So this packing bubble squish data actually turned out to be far more interesting than I initially expected, all from the happy accident of sending noisy load cell data to a Node-RED graph just because it was easily available. If I had to write my own code to graph the data, I probably would not have done it, and missed that interesting insight into the pressures of life as a packing bubble. This is a win for Node-RED.

The next challenge is to figure out how I could have captured, analyzed, and extracted that data programmatically. Human visual insight is very useful, but it requires that we think of the right way to graph data in a way that is useful. This is hard when we don’t necessarily know what we are looking for. I stumbled across this happy accident today, how might I make sure I don’t miss interesting insights tomorrow? Something to ponder…

In the meantime I have a more mundane question to answer: how do I maintain a record of work I’ve done in a Node-RED program?

Packing Bubble Squish Test

I wrote down my first impressions of Node-RED Dashboard, here I describe the project I used to explore and exercise my new tools in the Node-RED toolbox. It is a silly little thing that tests the squishiness of a plastic air bubble used as packing material. The bubble isn’t important, the main objective was to build my first nontrivial Node-RED flow and have it interface with my two recent hardware projects: the 3-axis motion control chassis of a retired Geeetech A10 printer, and the load cell kit mounted where the printer’s build plate used to be.

Both of these hardware components use USB connections that show up on the computer as serial ports, which made it easy to interface with Node-RED via node-red-node-serialport. Once installed, I had three new nodes on my palette. “Serial in” is what I needed to read the stream of data coming in from the Arduino handling the HX711 reading the load cell strain gauges. “Serial request” is what I needed for bidirectional data transfer with the 3D printer control board: sending G-code commands and reading status like position. (The third one, “Serial out”, is not applicable to this project.)

To keep the project simple, my X- and Y-axis motion are hard coded and I only hooked up a few Dashboard buttons to control my 3D printer motion in the Z-axis. This allowed me to fine tune the height of the carriage. I added buttons to remember two set heights A and B, and a toggle to automatically cycle between those positions.

I set my plastic bubble test subject down on the platform and used my flow to make the machine repeatedly press down on the bubble. Pressure reported by the load cell is sent to a Node-RED chart to graph its change over time. I was more interested in the exercise than any real results, but the graph actually turned out to be interesting.

Fast and Easy UI via Node-RED Dashboard

In addition to JSONata, there’s another very important project that is technically not part of core Node-RED. But it is rare to see one without another, and I’m speaking of the Node-RED Dashboard add-on module.

Every single Node-RED project I’ve seen makes use of the Dashboard module to build a HTML-based user interface. Which was why when I started learning Node-RED tutorials I was confused there were no mention of how to build an user interface. It took some time before I realized the Dashboard is not considered part of the core system and had to be installed separately. Once installed, there were additional nodes representing UI and additional editor interface. (Some UI to build UI…)

Once I finally realized my misconception and corrected it, I was able to build a functional interface in less than ten minutes, an amazingly short time for getting up and running under a new user interface construction system. Basic input controls like buttons and sliders, basic output controls like gauges and charts, they all worked just by connecting nodes to feed data.

However, the layout options are fairly limited. While it is extremely easy to build something closely resembling what I had in mind, I see no way to precisely adjust layout details. The rest of Node-RED reminds me of snapping LEGO pieces together, but the Dashboard exemplified the feeling: I can quickly snap together something that resembles the shape I had in mind, but if a distance is not an even number of LEGO stud spacing, I’m flat out of luck.

But even if I don’t see options for custom layout, I found instructions for building my own display widgets. Node-RED Dashboard is built on AngularJS a.k.a. Angular 1.x. I’m not sure I want to invest the time to learn AngularJS now. I can probably pick up enough AngularJS to do a custom widget if all I want is that single widget and reuse everything else. But AngularJS is currently on long-term support status receiving only security updates. Fortunately the fact Node-RED Dashboard is an add-on means people can (and have) built their own dashboard using other UI frameworks by hooking into the same extensibility mechanisms used by Dashboard. So if I want precise control over layout or other custom mechanism, I can do that while still using Node-RED as the underlying engine. I’m impressed we have that extremely powerful option.

But those dreams of grand expansion and customization are for the future. Right now I still need to build experience working with the system, which means putting it to work on a simple test project.

JSONata Reduces Need For Node-RED Function Nodes

Node-RED is built on sending messages from one node to the next. While there are some recommended best practices, it is really a wide-open system giving users freedom on how to structure the relationship between their nodes. As for the messages, they are JavaScript objects and there’s a convention to structuring them within a Node-RED project. While I can always fall back to writing JavaScript to work with messages, for the most part I don’t have to. Between the Cookbook and the Best Practices document, there are many methods to accomplish some commodity programming flow control tasks without dropping to a JavaScript function node.

But the built-in nodes have limited data manipulation capabilities. So I thought anything that requires actual data manipulation requires dropping to code. I was pleasantly surprised to find I was wrong: simple text changes and similar manipulation can be done without dropping to a JavaScript function node, they can be done with a JavaScript-based query and transformation language called JSONata.

JSONata appears to be an independent project not directly related to Node-RED, but it fits in very well with how Node-RED works and thus widely supported by many property fields of nodes. Any JavaScript string or data manipulation that could fit within a line or two of code can probably be expressed in JSOSNata, and thus accomplished directly in a standard Node-RED field without requiring dragging in a fully fledged JavaScript function node.

JSONata is yet another novelty in my exploration of Node-RED. I can vaguely sense this can be tremendously powerful, and I look forward to experimenting with this capability for future projects. But there’s another technically-not-core Node-RED feature that I will definitely be using, and that is the Node-RED Dashboard.

Node-RED Recommended Best Practices

Learning a new programming language, especially one with an entirely different paradigm, is confusing enough without having to worry about best practices. But after climbing enough of the learning curve, things quickly start getting chaotic and a little structure would help. I found this to be even more true for flow-based programming in Node-RED because a growing collection of nodes and wires connecting them can quickly grow into spaghetti code in a more literal sense than what I’ve been used to. A blank and pristine Node-RED flow doesn’t stay neat and pristine for long.

Fortunately, Node-RED documentation has a section called Developing Flows to help poor lost souls like me. It collects some basic recommendations for keeping flows manageable. And just like the Cookbook, it made more sense for me to read them after getting some hands-on experience building a bird’s nest of crossed wires and scattered nodes.

I felt sheepish to learn that I can have multiple tabs in the editor workspace. I should have noticed up top with a shape surrounding the name “Flow 1” and the plus sign to its right, but I had missed it completely. Each tab is a flow and when the project is deployed, all tabs (flows) execute simultaneously in parallel in response to their respective messages. This inherent parallelism does indeed remind me of LabVIEW.

Obviously multiple tabs make it easy to have unrelated features running in parallel, but what if they need to communicate with each other? That’s where I can use the link-in and link-out nodes. The set of link-in and link-out nodes with matching names act as wires connecting those nodes together.

They can also be used to declutter wires within a single flow. They still act the same way, but when one of the nodes is clicked, a dotted line representing the wire is visible on screen to make it easy to trace flow. Once unselected, the dotted line disappears.

A set of nodes can be combined together into a single “subflow“. In addition to decluttering, a subflow also aids in code reusability because a single subflow can be used multiple times in other flows and they all execute independently.

And finally, multiple adjacent nodes within a flow can be associated together as a group. The most obvious result is visually identifying the group as related. The editor also allows moving all the nodes in the group as a single unit. Beyond that, I don’t know if there are functional effects to a group, but if so I’m sure I’ll find them soon.

As an ignorant beginner, my first thought to flow organization most closely resembled groups. Which is why I was a little surprised to read it was added only very recently in 1.1.0. But once I read through the best practices recommendation in this Developing Flows section, I learned of all the other aspects of keeping flows organized, and I can see why groups hadn’t been as critical as I originally thought.

On the other side of the coin, as I explored Node-RED I found several other software modules that are deeply ingrained in many Node-RED projects, but aren’t technically a part of Node-RED. JSONata is one of these modules.

Node-RED Cookbook Was More Useful After Some Experience

The Node-RED User’s Guide helped me get started as a beginner on a few experimental flows of my own, slowly venturing beyond the comfort of JavaScript functions. But it was a constant process of going back and forth between my flow (which is not working) and the user’s guide to understand what I was doing wrong. I had fully expected this and, as far as beginner learning curves go, Node-RED is not bad.

On the Node-RED Documentation page, a peer of the User’s Guide is the Cookbook. I thought its subtitle “Recipes to help you get things done with Node-RED” was promising, but as a beginner I could not make use of it. It listed some common tasks and how to perform those tasks, but they were described in Node-RED terminology (‘message’ ‘flow’) which I as a beginner had yet to grasp. I couldn’t use the recipes when I didn’t even understand the description of the result.

Continuing the food preparation analogy: If I didn’t understand what “Beef Wellington” was, I wouldn’t know if I wanted to cook it, or be able to find the recipe in the book.

So to make use of the Node-RED cookbook, I had to first understand what the terms mean. Not just the formal definition, but actually seeing them in practice and trying to use them a few times on my own. After a few hours of Node-RED exploration I reached that point, and the Node-RED cookbook became a powerful resource.

I don’t know how the Node-RED Cookbook could make this any easier. It’s the kind of thing that was opaque to me as a beginner, but once I understood, everything looks easily obvious in hindsight. I stare at the cookbook descriptions now, and I don’t understand how I couldn’t comprehend the same words just a few days ago. I wish I could articulate something useful and contribute to help the next wave of beginners, because that would be amazing. But for now I can only be the beginner, consuming existing content like a Best Practices guide.

Node-RED Function Nodes Are A Comforting Fallback

Node-RED beginners like myself are given some hand-holding through two tutorials, creatively titled Creating Your First Flow and Creating Your Second Flow. After that, we are dropped into the User’s Guide for more information. The Using Node-RED section of that page covers fundamentals to get up to speed on how to work in a Node-RED project. Within that section, the page I found most instructive and informative is Using the Function Node.

Part of this might just be familiarity. A function node is a node that encapsulates a JavaScript function for doing whatever the author can write JavaScript to do. Because I’m familiar with languages like C and Python, I’m comfortable with the mentality of writing functions in source code to do what I have in mind. So seeing the function node and all I can do within it is comforting, like seeing a familiar face in a new crowd.

And just as in real life, there will be some level of temptation to stay in the comfort zone. It is probably possible to write any Node-RED program with just three nodes: an input node, a single Function node with a lot of JavaScript code, and an output node.

But writing all my logic in a single JavaScript function node would be ignoring the power of the platform. Flows allows me to lay out my program not in terms of functions calling one another, but in terms of messages flowing from one node to the next. Each node is an encapsulated representation of a feature, and each message is a piece of information that was generated from one node to inform another node on what to do next.

This is a different mentality, and it’ll probably take a bit of practice for me to rearrange my thinking to take advantage of the power of the platform. But while that transition is taking place, I expect to get occasionally stuck. But I know I can unblock myself by resorting to little pieces of JavaScript programming inside a big data flow program, and that’s a good confidence builder for me to proceed building some hands-on experience with Node-RED. I needed that experience before I could understand additional Node-RED resources like the Cookbook.

New Exploration: Node-RED

While researching LabVIEW earlier, I came across several forum threads from people asking if there’s a cheaper alternative. I haven’t come across any answers for a direct free open source competitor, but a few people did mention that LabVIEW’s data flow style of programming had some superficial similarities to Node-RED. With the caveat they are very different software packages targeting different audiences.

Still, it sounded interesting to look into. This was reinforced when I saw Node-RED in a different context. An enthusiastic overview on Hackaday, with a focus on home automation processing data distributed via MQTT. My current project is focused on a single machine and not distributed across many network nodes, so I’m not going to worry about MQTT for the time being, but the promise of an easy way to consume, process, and visualize data is quite alluring. I’ll use my newly assembled load cell as a data source and learn how to integrate it with Node-RED.

But before that can happen, I need to install Node-RED and run through the beginner tutorials. There are many options but the easiest way for me is to install Node-RED is a community-contributed plugin for FreeNAS. This gives me an one-click procedure to install Node-RED into a FreeBSD Jail on my FreeNAS home server. And if I decide I didn’t like it, it is also a one-click cleanup.

The simplicity of setup, unfortunately, also means a lack of choice in basic configuration. For example, I have no idea how to properly secure a Node-RED instance installed in this manner.

But that’s not important right now, because the one-click plugin install has fulfilled the purpose of having Node-RED up and running for me to try beginner tutorials elaborately named “Create your first flow” and “Create your second flow“. Though partway through tutorials I got distracted by the National Weather Service web API.