I’ve successfully overlaid data from a AMG8833 thermal sensor on top of the Adafruit Memento camera viewfinder, turning it into a thermal camera. A very slow and sluggish thermal camera! Because my first draft was not written with performance in mind. To speed things up, I converted my thermal overlay to use TileGrid and take advantage of the compositing engine in Adafruit’s displayio library. In theory that should have been faster, but my attempt was not and I didn’t know how to debug it. I went looking for another approach and found MicroPython/CircuitPython has ported a subset of the powerful Python NumPy library as ulab.numpy. And furthermore, there was an example of using this library to interpolate AGM8833 8×8 data to a 15×15 grid in Adafruit learning guide Improved AMG8833 PyGamer Thermal Camera. Ah, this will do nicely.

Add Performance Timers

The first thing I got from that project is a reminder of an old lesson: I need to record timestamps during my processing so I know which part is slow. Otherwise I’m left with vague things like “TileGrid didn’t seem much faster”. I added several lines of code that recorded time.monotonic_ns() and a single line at the end of my loop that print() delta between those timestamps. Since the units are nanoseconds and these are slow operations, I get some very large numbers that were unwieldy to read. Instead of dividing these numbers by 1000, I right-shifted them by 10 bits to result in a division by 1024. The difference between “roughly microseconds” and “exactly microseconds” is not important right now and, in the spirit of performance, should be much faster.

Measure TileGrid Implementation

Here’s are four frames from my TileGrid implementation:

read 38087 scaled 3099 mapped 1789 grid 1728 blit 28223 refresh 360370 total 433296
read 37789 scaled 3099 mapped 1759 grid 1758 blit 30190 refresh 359803 total 434398
read 38713 scaled 3129 mapped 1788 grid 1729 blit 29683 refresh 362098 total 437140
read 38296 scaled 3129 mapped 1758 grid 1759 blit 29146 refresh 360579 total 434667

With a total of ~434ms per loop, this is just a bit over two frames per second. Here’s the breakdown on what those numbers meant:

• “read” is time consumed by reading 8×8 sensor data from AMG8833 sensor. This ~38ms is out of my control and unavoidable. It must occur for basic functionality of this thermal camera.
• “scaled” is the time spent normalizing 8×8 sensor data points between the maximum and minimum values read on this pass. This ~3ms is my code and I can try to improve it.
• “mapped” is the time spent translating normalized 8×8 sensor data into an index into my thermal color palette. This ~1.7ms is my code and I’m surprised it’s over half of “scaled” when it does far less work. Perhaps ~1.7ms is how long it takes CircuitPython to run through “for y in range(8): for x in range(8):” by itself no matter what else I do.
• “grid” is the time spent updating TileGrid indices to point to the color indices calculated in “mapped”. Since it’s basically the same as “mapped” I now know updating TileGrid indices do not immediately trigger any bitmap processing.
• “blit” copied OV5640 sensor data into a bitmap for compositing. This ~30ms is out of my control and unavoidable. It must occur for basic functionality of this thermal camera.
• “refresh” is where most of the time was spent. A massive ~360ms triggered by a single line of my code. This included pulling bitmap tiles based on TileGrid indices, rendering them to the TileGrid, compositing thermal overlay TileGrid on top of the OV5640 bitmap TileGrid, and finally send all of that out to the LCD.

Back to Bitmap

I don’t know why my TileGrid compositing consumed so much time. I’m probably doing something silly that crippled performance but I don’t know what it might be. And when it’s all triggered by a single line of my code, I don’t know how to break it down further. I will have to try something else.

---

https://github.com/Roger-random/circuitpython_tests/commit/1a62d8adbbeecf9d05ad79ff239906367fbfb440

By overlaying data from AMG8833 thermal sensor on top of the Adafruit Memento camera viewfinder, I’ve successfully turned it into a thermal camera. The bad news is all of my bitmap manipulation code runs very slowly, bogging the system down to roughly a single frame per second. I blame my habit or writing Python code as if I were writing C code. Running tight loops shuffling bits around is fine in C, but now the same approach is incurring a lot of Python runtime overhead.

As I understand Python, the correct approach is to utilize libraries to handle performance-critical operations. My Python code is supposed to convey what I want to happen at a higher level, and the library translates it into low-level native code that runs far faster. In this context I believed I needed CircuitPython displayio sprite compositing engine to assemble my thermal overlay instead of doing it myself.

The viewfinder image is pretty straightforward, loading OV5640 into a Bitmap which went into a TileGrid as a single full-screen entry. The fun part is the thermal overlay. I created a TileGrid of 8×8 tiles, matching thermal sensor output data points. I then created another bitmap in code corresponding to my range of thermal colors. I didn’t see any option for alpha blending in displayio and, as I believed it to be computationally expensive, I wanted to avoid doing that anyway. My palette bitmap is again a screen door of my thermal color alternating with the color marked as transparent so viewfinder image can show through.

In theory, this means every thermal sensor update only requires updating tile indices for my 8×8 TileGrid, and displayio will pull in the correct 30×30 pixel bitmap tile to use as a sprite rendering my 240×240 pixel thermal overlay. The underlying native code should execute this as native code memory operation far faster than my loop in Python setting bitmap pixels one by one.

I had high hopes, but I was hugely disappointed when it started running. My use of TileGrid did not make things faster, in fact it made things slower. What went wrong? My best hypothesis is that compositing tiles with transparent pixels incur more workload than I had assumed. I also considered whether I incurred color conversion overhead during compositing, but as documentation for displayio.Palette claimed: “Colors are transformed to the display’s format internally to save memory.” So in theory color conversion should have been done once during startup when I created the thermal color tiles, not during the performance-critical loop.

The upside of Python’s “offload details to libraries” approach is that I don’t have to understand a library’s internals to gain its benefits. But the corresponding downside is that when things go wrong, I can’t figure out why. I have no idea how to get insight into displayio internals to see what part of the pipeline is taking far longer than I expected. Perhaps I will eventually gain an intuition of what is quick versus what is computationally expensive to do in displayio, but today it is a bust and I have to try something else.

---

https://github.com/Roger-random/circuitpython_tests/commit/650f46e64bc08de9f8c1f451a4d18ea7021e92fb

The AMG8833 thermal sensor I taped to an Adafruit Memento camera is successfully communicating with the ESP32-S3 microcontroller running Memento, and I can start working on integrating data from both thermal and visual cameras.

Goal

Low resolution thermal data can be difficult to decipher, but overlaying low-resolution thermal data on top of high-resolution visual data helps provide context for interpretation. This is a technique used in commercial thermal imaging products. The most accessible devices are designed to plug into my cell phone and utilize the phone for power and display. For my Android phone, it’ll be something like this FLIR One unit.(*) I’ve thought about buying one but never did. Now I will try to build a lower-cost (though far less capable) DIY counterpart.

Precedence

For code functionality, there’s a useful precedence in Adafruit’s “Fancy Camera” sample application: it has a stop-motion animation mode which shows the previously captured frame on top of the current viewfinder frame. This allows aspiring stop-motion animators to see movement frame-to-frame before committing to a shot, but I want to try using its overlay mechanism for my purposes. On the source code side, this means following usage of the data objects last_frame and onionskin. They led me to bitmaptools.alphablend(). Performing alpha blending on a microcontroller is not fast, but it was a good enough starting point.

Drawing Thermal Overlay

Now that I’ve found a helper to blend the viewfinder image with my thermal data, I have to draw that thermal data. The small LCD on board Memento has a resolution of 240×240 pixels, and that divides neatly into 8×8 sensor resolution. Each sensor data point corresponds to a 30×30 pixel block of screen. Drawing solid squares was really, really slow. I opted to draw every third pixel vertically and horizontally, which means drawing a dot for every 3×3=9 pixels. This lent a screen door effect to the results that was, again, good enough as a starting point.

Thermal Color Spectrum

Commercial thermal cameras have established a convention for color spectrum representing thermal data. Black represents cold, blue is a bit warmer, then purple, red, orange, yellow, all the way to white representing the hottest portion of the picture. I started mapping out a series of RGB values before I noticed that spectrum is conveniently half of a HSV hue wheel. I went looking for a CircuitPython library for HSV color space and found FancyLED. Calling pack() gave me a representation in RGB888 format instead of the RGB565_SWAPPED format used by Memento LCD. I didn’t find an existing conversion utility, but I’m a C programmer and I’m comfortable writing my own bit manipulation routine. It’s not the fastest way to do this, but I only have to build my palette once upon startup so it’s not a concern for the performance-critical inner display loop.

    # Obtain hue from HSV spectrum, then convert to RGB with pack()
    rgb = fancy.CHSV(hue, saturation, value).pack()

    # Extract each color channel and drop lower bits
    red =   (rgb & 0xFF0000) >> 19
    green_h3 = (rgb & 0x00FF00) >> 13
    green_l3 = (rgb & 0x003800) >> 11
    blue =  (rgb & 0x0000FF) >> 3
    # Pack bits into RGB565_SWAPPED format
    rgb565_swapped = (red << 3) + (green_h3) + (green_l3 << 13) + (blue << 8)

Orientation

I was happy when when I saw my thermal color overlay on top of the viewfinder image, but the two sets of data didn’t match. I turned on my soldering iron for a point source of heat, and used that bright thermal dot to determine my thermal sensor orientation didn’t match visual camera orientation. That was easily fixed with a few adjustments to x/y coordinate mapping.

Field of View

Once the orientation lined up, I had expected to adjust the scale of thermal overlay so its field of view would match up with the visual camera’s field of view. To my surprise, they seem to match pretty well right off the bat. Of course, this was helped by AGM8833’s low resolution giving a lot of elbow room but I’m not going to complain about having to do less work!

Too Slow

At this point I had a first draft that did what I had set out to do: a thermal overlap on top of visual data. It was fun taking the camera around the house pointing at various things to see their thermal behavior. But I’m not done yet because it is very sluggish. I have plenty of room for improvement with performance optimization and I think TileGrid will help me.

---

(*) Disclosure: As an Amazon Associate I earn from qualifying purchases.

https://github.com/Roger-random/circuitpython_tests/commit/30e24717cad579a0cc05f4b381d5f637259fe4bb

I’ve taped an AMG8833 thermal sensor to the side of an Adafruit Memento camera, just a quick hack for mechanical attachment while I experiment. I want to get them to work together and show something interesting, which means I need to figure out the software side. Here were my initial bootstrap steps:

Boarding An Existing I2C Bus

The first test was to see if the device is properly visible on Adafruit Memento’s I2C bus. Adafruit sample code failed when it tried to create a I2C busio object because it was written with an implicit assumption the AMG8833 was the only I2C device present. When mounted on an Adafruit Memento, I need to grab the existing I2C object instead of creating a new one.

Data Elements Are Floating Point Celsius

One thing that I didn’t see explicitly called out (or I missed it) was the format of data points returned by calling Adafruit library. Many places explaining it will be an 8×8 list of list. That is, a Python list of 8 elements where each of those elements is a list of 8 data points. But what are the individual data points? After printing them to console I can finally see each data point is a floating point number representing temperature reading in Celsius.

I2C Operation On Every pixels Property Getter Call

One lesson I had to learn was to be careful how I call the pixels property getter. One of the sample code snippets had this:

for row in sensor.pixels:
    for temperature in row:
        ...[process temperature]...

And while I was experimenting, I wrote this code.

for y in range(8):
    for x in range(8):
        sensor.pixels[y][x]

Conceptually they are very similar, but at run time they are very different. Mine ran extremely slowly! Looking at the library source code revealed why: every call to the pixels property getter initiates an I2C operation to read the entire sensor array. In the first loop above, this happens once. The second loop with my “write Python like C code” habit meant doing that 64 times. Yeah, that would explain why it was slow. This was an easy mistake to fix, and it didn’t take much more effort before I had a working first draft.

I’ve been learning CircuitPython library implementation for Adafruit Memento (a.k.a. PyCamera) with the goal of doing something interesting beyond published examples. After brainstorming a few candidates, I decided to add an AMG8833 thermal sensor alongside. Mainly because I bought an Adafruit breakout board thinking it was neat and had yet to unwrap it. Today is the day.

According to my Adafruit order history, I bought this way back in 2018. Look at how faded that label has become. Over the past six years several project ideas had come and gone, the most recent one I can remember being an ESP32 web app like what I had built for the AS7341 spectral/color sensor. But none of them got far enough for me to unwrap this roll of pink bubble wrap.

Since the time I bought my sensor, Adafruit has added a higher-resolution thermal sensor to their product list. I told myself not to spend money on the newer fancier sensor until I actually use the one I had already bought. During this time Adafruit has also evolved the design, adding a STEMMA QT connector.

If I had one of the newer boards, I wouldn’t need to do any soldering. The Memento has a STEMMA QT port and this little cable would connect them together.

But since I have the old board, I cut the cable in half so I can solder wires and plug the other end into Memento.

For mechanical mounting, I thought I would use one of the existing mounting holes and bolt it to a Memento corner post. It’d be quick and easy but unfortunately the hole diameter is just a tiny bit too small for this idea to work.

With that idea foiled, my brain started thinking about alternate approaches that grew more and more elaborate. I didn’t want to invest in the time and effort because I didn’t even know if this idea would work. I taped it down for the sake of expedient experimentation until a proof-of-concept first draft is up and running. Time to start coding.