Putting Adafruit Uncanny Eyes on a Tube TV

I have extracted the NTSC color composite video generation code from rossumur’s ESP_8_BIT project, and tested it by showing a static image. The obvious next step is to put something on screen that moves, and my test case is Adafruit’s Uncanny Eyes sketch. This is a nifty little display I encountered on Adafruit’s Hallowing and used in a few projects since. But even though I’ve taken a peek at the code, I’ve yet to try running that code on a different piece of hardware as I’m doing now.

Originally written for a Teensy 3 and a particular display, Uncanny Eyes has grown to support more microcontrollers and display modules including those on the Hallowing. It is now quite a complicated beast, littered with #ifdef sections blocking out code to support one component or another. It shows all the evidence of a project that has grown too unwieldy to easily evolve, which is probably why Adafruit has split into separate repositories for more advanced versions. There’s one for Cortex-M4, another one for Raspberry Pi, and possibly others I haven’t seen. In classic and admirable Adafruit fashion, all of these have corresponding detailed guides. Here’s the page for the Cortex-M4 eyes, here’s the page for Pi Eyes, in addition to the Teensy/Hallowing M0 eyes that got me started on this path.

If my intent was to put together the best version I can on a TV, I would study all three variants. Read Adafruit documentation and review the code. But my intent is a quick proof of concept, so I pulled down the M0 version I was already familiar with from earlier study and started merrily hacking away. My objective was putting a moving eye on screen, and the key drawEye() method was easily adapted to write directly to my _lines frame buffer. This allowed me to cut out all code talking to a screen over SPI and such. This code had provisions for external interactivity such a joystick for controlling gaze direction, a button for blink, and a light sensor to adjust iris. I wanted a standalone demo and didn’t care about that, and thankfully the code had provisions for a standalone demo. All I had to do was make sure a few things were #define (or not #define)-ed. That left a few places that were inconvenient to configure purely from #define, so I deleted them entirely for the purpose of the demo. They’ll have to be fixed before interactivity can be restored in this code.

The code changes I had to make to enable this proof of concept is visible in this GitHub commit. It successfully put a moving eye on my tube TV on my ESP32 using a composite video cable, running the color NTSC composite video generation code I pulled out of rossumur’s ESP_8_BIT project. With the concept proven, I don’t intend to polish or refine it. This was just a crude test run to see if these two pieces would work together. I set this project aside and moved on to other lessons I wanted to learn.

[Code for this project is publicly available on GitHub]

Extracting ESP_8_BIT Sega Color Video

ESP_8_BIT implemented a system that could run one of three classic video game console emulators on an ESP32, but my interest is not in those games. My objective is to extract its capability to generate and output a NTSC composite video signal, and significantly, doing it in color without hardware modification. I chose the Sega emulator video generation code path for my project, since it had better color than the Nintendo emulator code path. The Atari emulator code path was not a candidate as it had stopped working on my ESP32 for memory allocation failures, caused by factors I don’t understand and might investigate later.

The objective of today’s project will mean deleting the majority of ESP_8_BIT. Most of the bulk were cut away quickly when I removed all three emulation engines along with their game ROMs. All three emulation engines had an adapter class that derived from a base emulator (Emu) class, and they were removed without issue thanks to the clean architecture separation. All lower level code interact with the emulators via the base class. With so many references to the base class sprinkled throughout the code, it took more finesse to find and cleanly remove them all.

Beyond the color video generation code that is my objective, there were various other system level services that I wanted to strip off to get to a minimum functional baseline. My first few projects will go without any of the following functionality: Bluetooth controller interface, infrared (IR) remote interface, and loading ROMs from on-board SPIFFS storage of ESP32. These are all fairly independent subsystems. Once I have built confidence in the base video code, and if I should need any of the other system services, I can add them back in one at a time.

That leaves a few items intertwined with video output routines. The easiest to remove were the portions specific to Atari and Nintendo emulators, clearly marked off with #ifdef statements. ESP_8_BIT creator rossumur explained that the color signal hack had an unfortunate side effect of interfering with standard ESP32 audio output peripherals, so as an alternate mechanism audio had to be generated via ESP32’s LEDC peripheral originally designed for controlling LED illumination. Since audio didn’t seem to work on my setup anyway, I removed that mechanism as well. The final bit of functionality I didn’t need was PAL support since I didn’t have a PAL TV. I started removing that code but changed my mind. The work to keep PAL looks to be fairly manageable, and it’s definitely a part of video signal generation. Plus, I want this project to be accessible to people with PAL screens. However, I have to remember to document the fact that I don’t have a PAL TV to test against, so if I inadvertently break PAL support with this work I wouldn’t know unless someone else tells me.

Another reason for choosing the Sega emulator code path is that its color palette is a mapping I know as RGB332. It is a way to encode color in eight bits: three bits for red, three for green, and two for blue. This mapping is more common than hardware-specific palettes of either Atari or Nintendo paths. I anticipate using RGB332 will make future projects much easier to manage.

As a practice run of the code that’s left after my surgery, I wrote a test to load a color image. The was a picture Emily took of her cat, cropped to 4:3 aspect ratio of old school tube TVs and then scaled to the 256×240 resolution of the target frame buffer. This results in a PNG file that looks vertically stretched due to the non-square pixels. Then I had to convert the image to RGB332 color. Since this is just a practice run, I wrote a quick script to extract the most significant bits in each color channel. This is an extremely crude way to downconvert image color lacking nice features like dithering. But image quality is not a high concern here, I just wanted to see a recognizable cat on screen. And I did! This successful static image test leads to the next test: put something on screen that moves.

[Result of this extraction project is publicly available on GitHub]

Observations on ESP_8_BIT Nintendo and Sega Colors

I have only scratched the surface of rossumur’s ESP_8_BIT project, but I got far enough for a quick test. I rendered the color palette available while in Nintendo emulation mode, and found that it roughly tracks with the palette shown on its corresponding Wikipedia page.

I say “roughly” because the colors seem a lot less vibrant than I would have expected from a video game. These colors are more pastel, or maybe like a watercolor painting. For whatever reason the color saturation wasn’t as high as I had expected. But judging color is a tricky thing, our vision system is not a precision instrument. What I needed was a reference point, and I found one with the OSD (on-screen display) for this TV’s menu system.

This menu is fairly primitive, using basic colors at full saturation. I can see the brilliant red I expected for Mario, alongside the brilliant green I expected for Luigi. These colors are more vibrant than what I got out of the Nintendo emulator palette. What causes this? I have no idea. NTSC colorburst math is voodoo magic to me, and rossumur’s description of how ESP_8_BIT performs it went high over my head. For the moment I choose to not worry about the saturation. I am thankful that I have colors at all as it is a far superior situation than earlier grayscale options.

At this point I felt confident enough to look at the Sega emulator code path, and was encouraged to see the table returned by its ntsc_palette() method had 256 entries instead of the 64 in the Nintendo path. Switching ESP_8_BIT into Sega mode, I can see a wider range of colors.

Not only are there four times the number of colors available, some of the saturation looks pretty good, too.

The red (partially truncated by overscan in the lower left) is much closer to the red seen on OSD, the yellow and blue looks pretty good, too. The green is still a bit weak, but I like what I see here. With its wider range of colors in its palette, I decided to focus on ESP_8_BIT Sega emulator code path from this point onwards.

Studying NES Section of ESP_8_BIT

I’m not sure why ESP_8_BIT Atari mode stopped working on my ESP32, but I wasn’t too bothered by it. My objective is to understand the NTSC color composite video signal generation code, which I assumed was shared by all three emulators of ESP_8_BIT so one should work as well as another. I switched the compile-time flag to Nintendo mode, saw that it started display on screen, and resumed my study.

The emu_loop() function indicated _lines was the frame buffer data passed from one of three emulators to the underlying video code. This was defined in video_out.h as uint8_t** and as its name implied, it is an array representing lines on screen. Each element of this array is a pointer to another array, storing uint8_t represent pixels on screen. This is a very flexible design that allowed variation in vertical and horizontal pixels, just by changing the size of one array or another. It also accommodates any extra data necessary for a line on screen, for example if there are padding requirements for byte alignment, without code changes. That said, ESP_8_BIT appears to use a fixed resolution of 256 pixels wide by 240 lines high: In video_out.h, video_init() always sets _active_lines to 240, and the for() loop inside blit() is hard-coded to 256.

Inside blit() I learned one of my assumptions was wrong. I thought video_out.h was common across all three emulators, but there were slight variations marked out by #ifdef sections. I confess I didn’t understand what exactly these different chunks of code did, but I could make two observations: (1) if I wanted to use this code in my own projects I only need one of the code paths, and (2) they have something to do with the color palette for each emulator.

With that lead, I started looking at the color palette. Each emulator has its own color palette, when running in NTSC mode (my target) it is returned by the aptly named ntsc_palette() function. This appears to be a fixed array which was interesting. 8-bit color was very limiting and one of the popular ways to get around this limitation was to use an optimized adaptive palette to give an impression of more colors than were actually available. Another reason I had expected to find variable palette was the technique of palette animation (a.k.a. color cycling) common in the 8-bit era. Neither technique would be possible with a fixed palette. On the upside, it simplifies my code comprehension task today so I can’t complain too much.

With some basic deductions in hand on how this code works, I wanted to do a quick exercise to test accuracy of my knowledge. I wrote a quick routine to fill the frame buffer with the entire range of 256 values, isolated into little blocks. This should give me a view of the range of colors available in the palette.

    for(int y = 0; y < 240; y++)
    {
      yMask = (((y>>4)<<4)&0xF0);
      for(int x = 0; x < 256; x++)
      {
        _lines[y][x] = ((x>>4)&0x0F) | yMask;
      }
    }

When I ran this in the Nintendo emulator mode, I got what appeared to be four repeats of the same set of color. Referencing back to Nintendo code path in video_out.h I can see it meshes. Color value was bitwise-AND with 0x3F meaning only the lower six bits of the color value are used. Cycling through 256 meant 64 colors repeated 4 times.

        case EMU_NES:
            mask = 0x3F;

I didn’t need four repeats of the same 64 colors, so I edited my test program a bit to leave just a single bar on screen. Now I can take a closer look at these colors generated by ESP_8_BIT.

ESP_8_BIT Atari Mode Mysteriously Stopped Working

I was interested in getting my hands on some code that could generate color NTSC composite video without esoteric hardware hacking, and rossumur’s ESP_8_BIT promised to do that. As soon as I had it loaded on my own ESP32 and turned on my old tube TV with composite input, I could see it delivered on that promise. Everything else in this project would just be icing on the cake.

Sadly I had a lot of trouble getting the rest of ESP_8_BIT to work for me, so there were precious little icing for me. The first problem I encountered was that I didn’t have audio. This was expected for the Atari logo screen, but I checked the ESP_8_BIT YouTube video and saw there was supposed to be sound when the robot came on screen. Mine stayed silent, darn.

The next step was to get an input device working. ESP_8_BIT included a Bluetooth HID stack to interface with Bluetooth peripherals. One of the options is a Bluetooth Classic (not BLE) keyboard, and I had a Microsoft Wedge Mobile Keyboard on hand for this test. This is a keyboard I knew worked for my Bluetooth-enabled computers as well as an Apple iPad, so I did not expect problems. Unfortunately after going through the ESP_8_BIT pairing procedure, my ESP32 went into an endless reboot loop. As an alternate to Bluetooth, ESP_8_BIT also supported some infrared controllers, but I didn’t have any compatible device on hand. So unless I’m willing to buy an IR controller or buy another Bluetooth keyboard, it appears that there will be no ESP_8_BIT gaming for me.

There was one other mystery. This initial test pass was made several weeks ago. Afterwards, I set down ESP_8_BIT to be explored later. I worked on a few other projects (including Micro Sawppy rover) in the meantime. When I returned to ESP_8_BIT, I could no longer run the Atari emulator. It now enters a failure reboot loop with an error of insufficient memory. Not by a lot, either: the largest available segment of memory was only a few hundred bytes short of what’s requested.

esp_8_bit

mounting spiffs (will take ~15 seconds if formatting for the first time)....
... mounted in 99 ms
frame_time:0 drawn:1 displayed:0 blit_ticks:0->0, isr time:0.00%
emu_task atari800 running on core 0 at 240000000mhz
MALLOC32 235820 free, 113792 biggest, allocating Screen_atari:92160
MALLOC32 allocation of Screen_atari:92160 3FFE4374
MALLOC32 143644 free, 64864 biggest, allocating MEMORY_mem:65540
MALLOC32 FAILED allocation of MEMORY_mem:65540!!!!####################
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1216
ho 0 tail 12 room 4
load:0x40078000,len:10944
load:0x40080400,len:6388
entry 0x400806b4

I hadn’t touched the code, and the hardware was only changed in that I actually have it on a circuit board this time. So I’m at a loss what might have caused this difference. One hypothesis is that an underlying Arduino library had been updated and took a little more memory than it used to, leaving not quite enough for the Atari emulator. Another hypothesis is that my Micro Sawppy code left something behind on this ESP32, impacting behavior of ESP_8_BIT.

Without more data or knowledge I couldn’t track down the source of this problem. So with the Atari emulator out of commission, my study course moved on to the Nintendo emulator for me to gain that knowledge.

NTSC Color Composite Video From ESP_8_BIT by rossumur

I’ve buried my head in remote-control vehicles for a while now, most recently writing code for my own Sawppy rover and looking over code written by others. Rover fatigue is setting in, so I’m going to look at a different topic as a change of pace.

About a year and a half ago Emily and I had a study session examining composite video signals. Using an oscilloscope to compare and contrast between different signal sources. Our reference was a commercial video camera, and we compared them against outputs generated by various microcontroller libraries. Two that run on ATmega328 Arduino, and one running on ESP32.

One thing they all had in common was that they were all in grayscale. The extra signal required for color NTSC composite video were beyond the reach of stock hardware. People have done some very creative things with hacking on oscillators to change operating frequencies, but even with such hacking the capability was quite limited. We thought color would be out of reach except by using chips with dedicated hardware such as the Parallax Propeller chip.

I was so happy to learn we were wrong! GitHub user rossumur found a way to generate color NTSC composite video signal from an ESP32 on stock unmodified hardware. The clever video hack was done using, of all things, an audio peripheral called the Phase Locked Loop. This color video capability formed the basis of an 8-bit video game console project called ESP_8_BIT, and I had to take a closer look.

For most projects I find online, I spend time researching the code before I bother investing equipment. But this project had such minimal hardware requirements that I decided to build a test unit first. I didn’t even need a circuit board: the video pin can be wired straight through and the audio pin only required a resistor and a capacitor. A few wires on a 0.1″ pitch connector and composite video jacks salvaged from an old TV were all I needed to install those parts on an ESP32 dev board.

Compared to the effort of wiring things up to my ESP32 dev board, getting my composite video screen actually took more effort. I had an old tube TV gathering dust sitting under a lot of other stuff, and it took more time to dig it up than to perform my ESP32 wiring. But once everything is plugged in and turned on, I could see the Atari logo on screen. And most importantly: it was in color! I’m going to follow this rainbow to the pot of ESP32 coding gold at the end, even if this adventure had an inauspicious start.

Windows Phone Debug Tools Rode Into Sunset

Organizing and commenting code for my Sawppy Rover ESP32 control project had two sides: the server-side code running on the ESP32 and the client-side code running in a web browser. The MIT license text and general comments went well, but when I tried to improve error handling, I ran into a serious problem with the old Internet Explorer browser built into Windows Phone 8.1: I have no debugging tools.

My only indication was that the page failed to load and all I saw was a blank screen. There were no error messages displayed onscreen, and unlike desktop browsers I couldn’t even look at what’s been printed to console.log(). Just a blank nothing. I backtracked through my code changes and eventually traced it down to a bit of error handling code I added to the JavaScript file.

try {
  // Do potentially problematic thing
} catch {
  // Use alternate approach instead
}

This worked in desktop browsers, this was also accepted in my modern Android phone’s Chrome browser, but it was treated as an error in Internet Explorer. Even though I didn’t care about the specific error thrown, IE didn’t permit omitting the specifier. The following syntax was required:

try {
  // Do potentially problematic thing
} catch(err) {
  // Use alternate approach instead
}

This code change was not, in itself, a big deal. But it took an hour of trial-and-error to find the location of the error (no feedback = no line number!) and figure out what the browser would tolerate. During this time I was operating blind with only a binary “blank screen or not” as my feedback mechanism. I need better tools if I am to tackle more challenging JavaScript programming on Windows Phone IE.

Unfortunately, almost all of the debugging resources for Windows Phone platform have disappeared. Microsoft’s own Visual Studio IDE — formerly the home of Windows Phone app development — don’t even mention the platform at all in its “Mobile Development” feature page. A promising resource titled Diagnosing Mobile Website Issues on Windows Phone 8.1 with Visual Studio (published in 2014) said to best tool to use is the Windows Phone emulator because it was easier. Avoiding all the hoops one must jump through to put a physical Windows Phone device in developer mode for debugging. Today it’s not just a matter of “easier” since the latter is outright impossible: the Windows Phone developer portal has been shut down and I can no longer put any of my devices into developer mode.

But perhaps they’re both equally impossible, as the Windows Phone emulator is no longer an option for installation in Visual Studio 2019. A search for related pages led me to Mobile Apps & Sites with ASP.NET (published in 2011) whose section had a promising link “Simulate Popular Mobile Devices for Testing“. But that link is no longer valid, clicking it merely redirects back to the Mobile Apps & Sites with ASP.NET page. Many search boxes later, I eventually found what claims to be the Windows Phone emulator as a standalone download. I did not try to download or install it because at that point I was no longer interested.

I aborted my intention to organize my browser JavaScript code. Right now everything sits as peers at top-level and globally accessible. I had intended to reorganize the code into three classes: One handles drawing, one handles user input (pointer events), and the third handles server communications (websocket). However It looks like Internet Explorer never supported the official JavaScript class mechanism. I can probably hack up something similar, JavaScript is flexible like that. People have been hacking up class-like constructs in JS long before the official keyword was adopted. But to do that I need debugging tools for when I run into inevitable problems. Doing it without even the ability to see syntax errors or console.log() is masochistic self-punishment and I am not interested in inflicting that upon myself. No debug tool, no reorganization. I will add comments but the code structure will stay as-is.

This frustrating debugging session sapped my enthusiasm for working on Sawppy rover ESP32 control code. But before I go work on something else for a change of pace, I wanted to get a basic voltage monitoring system up and running.

[Code for this project is publicly available on GitHub]

Cleaning Up And Commenting Sawppy Rover ESP32 Code

The rover is now up and running on its own, independent of my home network. I felt this was a good time to pause feature development of Sawppy ESP32 control software. From an user experience viewpoint, this software is already in better shape than my SGVHAK rover software. Several of the biggest problems I’ve discovered since our local rovers started running have been solved. I feel pretty good about taking this rover out to play. More confident, in fact, than rovers running my earlier software.

Before I turn my attention elsewhere, though, I need to pay off some technical debt in the areas of code organization and code commenting. I know if I don’t do it now, it’ll never happen. Especially the comments! I’m very likely to forget if I don’t write them down now while they’re fresh in my mind. The code organization needed to be tamed because there were a lot of haphazard edits as I experimented and learned how to do things in two new environment simultaneously: ESP32 and JavaScript. The haphazard nature was not necessarily out of negligence but out of ignorance as I had no idea what I was doing. Another factor was that I copied and pasted a lot of Espressif sample code (especially in the WiFi area) and I had instances of direct copied text “EXAMPLE” that needed to be removed.

I also spent time improving code robustness with error handling code paths. And if errors are not handled, they should at least be reported. I have many unhappy memories trying to debug some problem that made no sense, and eventually traced it to a failed API call upstream whose error code was never reported. Here I found Espressif’s own ESP_LOG AND ESP_ERROR_CHECK macros to be quite handy. Some were carried over directly from sample code. I’ve already started using ESP_ERROR_CHECK during experimentation, because again I didn’t have robust error handling in place at the time but I wanted to know immediately if something fails.

Since this was my first nontrivial ESP32 project, I don’t think it’s a good idea for anyone to use and copy my amateurish code. But if they want to, they should be allowed to. Thus another bulk edit added the MIT license prefix to all of my source files. Some of them had pieces copied from Espressif example, but those were released to public domain so I understand there would be no licensing conflict.

While adding error handling code, though, I wasted an hour to a minor code edit because I didn’t have debug tools.

[Code for this project is publicly available on GitHub]

Sawppy Rover Independence with ESP32 Access Point

I wanted to make sure I had good visual indication of control client disconnect before the next task on my micro Sawppy rover ESP32 brain: create its own little WiFi network. This is something I configured the Raspberry Pi to do on SGVHAK rover and Sawppy V1. (Before upgrading Sawppy to use a commercial router with 5GHz capability and much longer range.)

All of my development to date have been done with the ESP32 logged on to my home network. This helps with my debugging, but if I take this rover away from home, I obviously won’t have my home network. I need to activate this ESP32’s ability to act as its own WiFi access point, a task I had expected to be more complex than setting up the ESP32 as a client (station mode) but I was wrong. The Espressif example code for software-based access point (softAP) mode was actually shorter than its station mode counterpart!

It’s shorter because it was missing a lot of the functionality I had expected to see. My previous experience with ESP32 acting as its own access point had been with products like Pixelblaze, but what I saw was actually a separate WiFi manager module built on top of ESP32’s softAP foundation. That’s where the sophisticated capabilities like captive portal configuration are implemented.

This isn’t a big hinderance at the moment. I might not have automatic forwarding with captive portal, but it’s easy enough to remember to type in the default address for the ESP32 on its own network. (http://192.168.4.1) On the server side, I had to subscribe to a different event to start the HTTP server. In station mode, IP_EVENT_STA_GOT_IP signals that we’re connected to the access point and it has assigned an IP address. This doesn’t apply when the ESP32 is itself the access point. So instead I listen for a control client to connect with WIFI_EVENT_AP_STACONNECTED, and launch the HTTP server at that point. The two events are not analogous, but are close enough for the purpose micro rover control. Now it can roam without requiring my home WiFi network.

Sometime in the future I’ll investigate integrating one of those WiFi manager modules so Sawppy rover users can have the captive portal and all of those nice features. A quick round of web searching found this one by Tony Pottier, with evidence of several more out in circulation. These might be nice features to add later, but right now I should clean up the mess I have made.

[Code for this project is publicly accessible on GitHub.]

Micro Sawppy Beta 3 Running With HTML Control

After I established websocket communication between an ESP32 server and a phone browser JavaScript client, I needed to translate that data into my rover’s joystick command message modeled after ROS joy_msg. For HTML control, I decided to send data as JSON. This is not the most bandwidth-efficient format. In theory I could encode everything into binary with two signed 8-bit integers. One byte for speed and one byte for steering is all I really need. However I have ambition for more complex communication in the future, thus JSON’s tolerance for extra fields has merit from the perspective of forward compatibility.

Of course, that meant I had to parse my simple JSON on the ESP32 server. The first rule of writing my own parser is: DON’T. It’s a recurring theme in software development: every time someone thinks “Oh I can just whip up a quick parser and it’ll be fine” they have been wrong. Fortunately Espressif packaged the open source cJSON library in ESP-IDF and it is as easy as adding #include "cJSON.h" to pull it into my project. Using it to extract steering and speed input data from my JSON, I could translate that into a joy_msg data structure for posting to the joystick queue. And the little rover is up and running on HTML control!

The biggest advantage of using ESP32’s WiFi capability to turn my old Nokia Lumia 920 phone into a handheld controller is cost. Most people I know already have a touchscreen phone with a browser, many of whom actually own several with older retired phones gathering dust in a drawer somewhere. Certainly I do! And yeah I also have a Spektrum ground radio transmitter/receiver combo gathering dust, but that is far less typical.

Of course, there are tradeoffs. A real radio control transmitter unit has highly sensitive inputs and tactile feedback. It’s much easier to control my rover precisely when I have a large physical wheel to turn and a centering spring providing resistance depending on how far off center I am. I have some ideas on how to update the browser interface to make control more precise, but a touchscreen will never have the spring-loaded feedback. Having used a RC transmitter a few days before bringing up my HTML touch pad, I can really feel the difference in control precision. I understand why a lot of Sawppy rover builders went through the effort of adapting their RC gear to their rovers. It’s a tradeoff between cost and performance, and I want to leave both options open so each builder can decide what’s best for themselves. But that doesn’t mean I shouldn’t try to improve my HTML control precision.

[Code for this project is publicly available on GitHub.]

Shiny New ESP32 WebSocket Support

My ESP32 development board is now configured to act as HTTP server sending static files stored in SPIFFS. But my browser scripts trying to talk to the server via websocket would fail, so it’s time to implement that feature. When I did my initial research for technologies I could use in this project, there was a brief panic because I found a section in Espressif documentation for for websocket client, but there wasn’t a counterpart for websocket server! How can this be? A little more digging found that websocket server was part of the HTTP Server component and I felt confident enough to proceed.

That confident might have been a little misplaced. I wouldn’t have felt as confident if I had known websocket support was added very recently. According to GitHub it was introduced to ESP-IDF main branch on November 12, 2020. As a new feature, it was not enabled by default. I had to turn it on in my project with the CONFIG_HTTPD_WS_SUPPORT parameter, which was something I could update using the menuconfig tool.

Side note: since menuconfig was designed to run in a command prompt, using it from inside Visual Studio Code PlatformIO extension is a little different due to input handling of arrow keys. The workaround is that J and K can be used in place of up/down arrow keys.

Attempting to duplicate what I had running in my Node.js placeholder counterpart, I found one limitation with ESP32 websocket support: Using Node.js I could bind HTTP and WS protocols both to root URI, but this ESP32 stack requires a different URI for different protocols. This shouldn’t be a big deal to accommodate in my project but it does mean the ESP32 can’t be expected to emulate arbitrary websocket server configurations.

I also stumbled across a problem in example code, which calls httpd_ws_recv_frame with zero to get length, then allocating the appropriate sized buffer before calling it again to actually fetch data. Unfortunately it doesn’t work for me, the call fails with ESP_ERR_INVALID_SIZE instead of returning filled structure with length. My workaround is to reuse the same large buffer I allocated to serve HTML. I think I’ve tracked this down to a version mismatch. The “call with zero to get length” feature was only added recently. Apparently my version of ESP-IDF isn’t recent enough to have that feature in order to run the example code. Ah, the joys of running close to the leading edge. Still, teething problems of young APIs aside, it was enough to get my rover up and running.

[Code for this project is publicly available on GitHub.]

ESP32 HTTP Was Easy But Sending Files Need SPIFFS

I got my ESP32 on my local home WiFi network, but it doesn’t respond to anything yet. The next step is to write code so it could act as a static web server, sending files stored in ESP32 flash memory to clients requesting them via HTTP GET. The good news is that setting up a HTTP server was really easy, and I quickly got it up and running to serve a “Hello World” string hard-coded in my ESP32 source code.

I briefly considered calling this situation good enough and move on. I could embed my HTML/CSS/JavaScript files as hard-coded strings inside my C files. But doing so meant reviewing those files to make sure they don’t have conflicts with C string syntax, and that’s something I’d have to do every time I wanted to update any of those files. This is quite clumsy. What I really want is to keep those files to be served over HTTP separate from my C code, so that they could be updated independently of code and I don’t have to review them for C string incompatibility.

Doing this requires carving out a portion of an ESP32’s flash memory storage for a simple file storage system called SPIFFS. Allocation of flash storage is declared in a partition file, which is a list of comma-separated values (CSV) of partition information like size, offset, and assigned purpose. PlatformIO ESP-IDF project template includes a default partition file but it had no provision for a SPIFFS partition. I need to add one to my project to override that default. I found an example partition file as my starting point, copying it into my project and making only minor changes. (I didn’t need quite as large of a SPIFFS partition.) If I were using ESP-IDF directly, I believe I would use the menuconfig tool to point to my partition file. But as I’m using ESP-IDF indirectly via PlatformIO, I specified my partition file location using a parameter in the platformio.ini configuration file.

Once I have a partition, I need to put my files in it. Apparently it’s not as simple as using a USB flash drive, I have to build the entire data partition (like building a disk image for a PC) and upload the whole thing. There are ESP-IDF command line tools to build SPIFFS partition, I decided to go with PlatformIO route by specifying the platformio.ini parameter data-dir. I could point it at my Node.js working directory, which is great because that meant I only had one copy of these files in my code repository. Eliminating the task of keeping multiple copies in sync. From the PlatformIO UI I could then use “Platform/Build Filesystem Image” followed by “Platform/Upload Filesystem Image”. I haven’t figured out if this is necessarily separate from code update or if there’s a way to do both at the same time.

Putting files in SPIFFS isn’t useful until I have code to read those files. I followed the links to Espressif code example for reading and writing to SPIFFS. And ugh… I haven’t had to deal with old school C file input/output API in quite some time. I briefly considered the effort to keep things memory efficient by breaking these file I/O actions up into small pieces, but I don’t need that kind of sophistication just yet. In order to simplify code, and because I have RAM to spare at the moment, I allocated a large buffer so all the operations to read from SPIFFS and send via HTTP can be done in a single pass. Which worked well for sending data to the control client, but now I need to listen to control commands coming back.

[Code for this project is publicly available on GitHub.]

Notes On Getting ESP32 On WiFi

I’m working towards a minimal first draft of my Sawppy rover’s HTML-based control scheme. I tackled client side first because that side had more similarity to my earlier SGVHAK rover control project. I got a first draft by using a Node.js server on my desktop as a placeholder for the ESP32, now it’s time to start working on the real server code. The first step is to learn how to activate an ESP32’s WiFi capabilities in software.

There are many options on how to set up networking between an ESP32 and a browser running on a phone. I decided to start with connecting it as a node to my home network, which Espressif documentation calls “station mode“. I expect this to be easier to develop and debug, but obviously it won’t work for a rover roaming away from home. For that the ESP32 will have to be switched to “AP mode” acting as a WiFi access point. In order to accommodate this, I aim to structure my code so setting up a WiFi connection is its own separate FreeRTOS task, so that I can switch between station and AP (or maybe even more?) variants as needed at compile-time. If I ever need the option to switch between modes at runtime, I can integrate one of the available ESP32 WiFi managers.

To learn about setting up an ESP32 in station mode, Espressif’s written documentation is pretty thin. I had to do most of my learning from their station example, and thankfully it was indeed enough to get my ESP32 on my network. Converting the sample code to a FreeRTOS task I can incorporate into my rover project required more than 2 kilobyte of stack, which is what I had been using as a default. A bit of quick experimentation established 3 kilobytes are enough for the new WiFi task. Right now I’m not too worried about RAM footprint, so I won’t spend time pinning down exactly where between 2 and 3 it needs.

One item that I saw and copied without understanding, was a code block initializing an ESP32’s non-volatile storage (NVS) used to store information in key-value pairs. I can infer that ESP32 WiFi stack required some of those values, but I couldn’t find documentation on exactly what was needed. The more obvious WiFi-related pieces of information (like the SSID for my home network and associated password) are in my code, so it must be something less obvious but I don’t yet know what they are.

Events were an aspect of the sample code that confused me for a while. Eventually I figured out my confusion stemmed from the fact I got two items mixed up. One of them is an “Event Group“, which is a FreeRTOS mechanism used here so a piece of code could wait until WiFi initialization completes before they start doing network things. The other mechanism is an “Event Loop” which is an Espressif creation for ESP-IDF. They both had “event” in their name but were completely independent, which was why I got lost trying to understand how the two mechanisms interacted behind the scenes: the answer is that they don’t! So when reading the code it’s important to figure out which “event” we are talking about. The Espressif reference for WiFi initialization events, for example, refers to the event loop.

I’m happy I was successful in connecting an ESP32 to my home WiFi by following example code. Now that it is on the network, I need to make it do something useful.

[Code for this project is publicly available on GitHub.]

Micro Sawppy RC Input Via ESP32 RMT

As one of many options I wanted to offer on my micro Sawppy rover ESP32 brain, I wanted to teach it to understand servo motor signals sent by radio control receivers. I found an example online of a project doing a similar thing and read through their code to see how they did it. Cross checking with Espressif documentation on ESP32’s RMT peripheral, that precedent taught me how to configure RMT to read servo control PWM input.

When configured for signal receive, RMT watches the state of a digital signal and measure the time duration for on and off periods of signal pulses. There are a set of events that can be configured to trigger an interrupt service routine (ISR) for the application developer to pick up the measured data for processing. There are two challenges here. First is the general challenge of writing an ISR. There are constraints on what ISR code can do, and what APIs it can call. Violating such constraints will usually lead to mysterious hard-to-diagnose problems. The second challenge is specific to the ESP32 RMT peripheral, because the ISR code will have to know where to fetch data, which flags to check, and which flags to set or clear upon exit. Again doing this wrong usually results in weird behavior.

Fortunately, it’s not absolutely required to write my own ISR. Espressif provides a default ISR for handling RMT events and parses data into a defined format and communicated via a ring buffer, an Espressif extension to FreeRTOS. Espressif’s own RMT sample code to process infrared remote control signal uses this default ISR, and I decided to follow that precedent instead of diving into the challenge of writing my own ISR.

One of the RMT parameters is a clock divisor, stepping the default 80MHz sampling rate down to something that would work well for the signal at hand. I decided to use a divider of 80, so the signal is sampled at 1 MHz. This makes the math easier for me to understand in my head, because each RMT ‘tick’ is now one microsecond. The servo control signal pulse of 1 to 2 milliseconds maps cleanly to 1000 to 2000 microseconds. This is lower resolution than the divider of 10 used by my RC precedent, but it means I don’t have to multiply everything by eight in my head and having one thousand levels of differentiation should still be more than enough.

My first draft didn’t work because the default RMT ISR would never send data on the ring buffer. I know it is retrieving data and putting it on the ring buffer, because after a few seconds I would see “ring buffer full” error on the serial debug monitor. Eventually I figured out I misunderstood what “idle” meant in RMT documentation. I thought “idle” meant we lost the signal and need to go into some kind of recovery routine, so I set it to 32 milliseconds. This is wrong! “Idle” in this context actually meant the end of a single data signal of interest. Because servo control signals were arriving every ~16 milliseconds, the RMT idle threshold was never tripped and thus RMT thought data had not yet completed. The fix is to change idle threshold down to something far shorter than 16 milliseconds but comfortably longer than actual signal. Once I understood my problem and fixed it, RMT ISR started signaling data ready on the ring buffer, allowing me to retrieve servo control durations and translate that to my micro Sawppy joystick messages. Once that translation layer is in place, the little rover is no longer tied to a wired joystick!

Notes on ESP32 RMT Peripheral For Receiving RC PWM

I want my micro Sawppy rover’s ESP32 brain to be capable of accepting control input from multiple sources. One of them will be my Spektrum SR300 radio control receiver. I examined its output control signal with a Saleae Logic 8 to make sure I understood what I will be working with, and the next step is to figure out how to interpret those signals with an ESP32.

With no shortage of hardware peripherals, including multiple options to generate PWM signals for servo control, I was confident an ESP had something to accept input. A bit of research quickly pointed me to its RMT peripheral, which was primarily designed to interface with infrared remote controls like our living room TV. But right in the documentation it also said “Due to flexibility of RMT module, the driver can also be used to generate or receive many other types of signals” and it looks like interpreting RC PWM is one of those “many other types of signals.”

Espressif pointed to several examples, but most of them only demonstrated how to send signals. Only one demonstrated how to interpret received signals, and it was for infrared remote control. I went out to the internet community of ESP32 users to find an example interpreting RC PWM signals, and I first came across this Reddit question, which was answered by [Justin Ong] with “I’ve done it and here’s my code.” I love forum threads with answers like this.

I didn’t try to download and run this code, but it was a valuable resource for me to cross-reference against Espressif documentation. From this I understood how RMT is used to sample the input signal at a particular rate, looking for rising and falling edges and reporting the time that has transpired between those edges. Once all the pieces came together in my head I understood why it is the perfect tool for interpreting pulse-width signals for hobby servo control.

After reading this example I think I have a good grasp of RMT configuration parameters and how the code interpreted the measured time durations. The part I’m still fuzzy about is the ISR (interrupt service routine.) This is the code that actually responds to an rising or falling edge event, and retrieves data from RMT peripheral hardware registers for later processing elsewhere. There were several pieces of bitwise manipulation code that I struggled to follow due to its tie with ESP32 hardware. This is not easy for people to grasp, the code comments referenced this one out of many forum threads from people confused about what they are expected to do. I see this as a warning flag that it’s very easy to make mistakes writing a RMT ISR, so I was glad to learn that I didn’t have to.

Micro Sawppy Beta 3 USB Access Port

When my little rover Micro Sawppy Beta 3 (MSB3) is running about, its brain built from an ESP32 dev module will be running from battery power. But when I’m updating its software via USB, I have to disconnect the power line to ensure it stays separate from my computer’s USB power line. I originally intended this disconnect to be done by removing a jumper. But after the jumper proved to be too buried to be easily removed, I made a quick hack to change it into a wire. Next step: make the USB port itself easier to access, because it is very much buried in there and hard to access.

I built MSB3’s body before knowing exactly what this circuit board will look like. My original intent is to make the bottom of the tray removable, held only by a few clips. Once those clips were removed I could drop the tray and access the USB port.

Unfortunately, this meant the circuit board is dangling by all the wheel motor and steering servo connections, which meant I’m constantly running the risk of accidentally pulling some wires. Either partially, resulting in an annoying intermittent connection. Or completely, resulting in a loss of functionality or possibly worse with power flowing where it isn’t supposed to. I didn’t like that risk before, and I’m certainly not happy about it now. What I want is a way to access the micro USB port without risking any unplanned wire removal, and that port is sitting behind this unbroken plastic face.

Since this is merely a prototype, there’s no reason why that plastic face must remain unbroken. So I pulled out my drill and started drilling. Starting with a small hole to create a pilot and verify I have the correct location, and gradually working my way towards larger diameter drill bits.

One of the larger drill bits caught and split the case along a printed layer line, but I could still keep things together with my clips so there is no functional problem with the new cracks. It is merely a bit of cosmetic embarrassment, which I can tolerate for the benefit of easy access to my ESP32 dev kit’s micro USB port. There are more things I would like to have in a micro Sawppy rover control board, but this is enough for me to continue working.

Provision for Micro Sawppy Voltage Monitoring

Recent experience iterating through Micro Sawppy prototypes made it clear I underestimated the task of designing a power supply scheme to fit all of my objectives. My blind spot came from the fact Sawppy V1 was up and running with a very simple scheme for power. I had a two-cell lithium-polymer battery pack, and almost all the components on board were happy to take power directly from that battery and perform their own internal voltage regulation. The only exception was the Raspberry Pi 3 on board, to which I attached a MP1584 buck converter to supply a consistent five volts. It was very little effort to get Sawppy V1 working, so I had the misconception power schemes are easy! They are not.

In order to meet my cost objectives for Micro Sawppy, I switched to different components. These simpler components were far more particular about their power supply, so I had to take on more of the power considerations that were previously a feature built in to more expensive parts. We are now at the point where I think I need to pull the rover’s ESP32 brain into the discussion. The motivation here are the six TT gearboxes and their corresponding motors, officially rated for operation up to 6 volts but can tolerate brief periods above that.

In order to stay below that maximum, the ESP32 can limit its maximum motor control PWM duty cycle sent to the DRV8833 motor control ICs. In an ideal world, if I had a 7.4V power supply, I should be OK as long as I limit PWM duty cycle to no more than (6 / 7.4) = 81%. But this voltage value would change as the battery depletes. When the battery is fully charged at 8.4V, 81% would delivery too much power. And as it approaches depletion of 7V, 81% would be too low to obtain 6V output. What I really want is for the PWM duty cycle to be dynamically adjusted based on battery voltage.

Conveniently, the current pin assignment for ESP32 dev kit still has one input pin open and available for use. So I soldered a pair of resistors to that pin. An 1 MOhm resistor to the battery voltage pane, and a 100 KOhm resistor to ground. This gives me an 11:1 voltage divider which I should be able to read with one of ESP32’s ADC (analog-to-digital conversion) peripherals. This provision will still need corresponding software work before it’ll do anything useful. But if it doesn’t work, it should be pretty easy to clip those resistors off.

The primary objective for voltage monitoring is to dynamically adjust PWM duty cycle in order to maintain rover performance as the battery discharges. Secondary objective is to let the ESP32 send out a low battery alert if the battery is low. Sawppy V1 used an external battery voltage alarm (*) but if I can incorporate that feature into ESP32 software it’ll cut down on parts cost. At the very least, I would like to put the rover into limp mode if the battery voltage drops below a threshold, which would be a feature missing from Sawppy V1.

I expect that battery voltage drop would make the motors unreliable well before it makes the ESP32 unreliable, but as my breadboard test showed, that is still possible. So another potential work item on the to-do list is to enable ESP32 brownout detection capability and recognize when battery voltage is dragged down by rover motors.

Configurable Micro Sawppy Servo Power Supply

My first soldered control board for Micro Sawppy was a huge mess of wires, most of which were related to power distribution. I am confident future revisions will improve, partly from experience and partly from features that I won’t need anymore. One of them is visible in the center of the title picture: a jumper by a MP1584 buck converter voltage regulator. It allowed me to switch between powering the servo with that buck converter and powering the servo directly from battery input power.

Earlier I had determined that four AA alkaline batteries had the right voltage for SG90 micro servos, but that voltage would sag significantly under the load of TT gearmotors driving six wheels. Adding another battery would destroy the servo when the system is unloaded. I knew there were a lot of power losses on my earlier breadboard-based prototype. I thought a soldered board would be a more accurate test. But while it may have made some difference, it was not enough to help Micro Sawppy run reliably on four AA batteries.

If Micro Sawppy is to be powered by alkaline batteries and still avoid using a voltage regulator for servos, I would have to move upscale to higher amperage C or D batteries. I also contemplated the idea of trying one of those large rectangular 6V lantern batteries but they all share the problem of availability. Alkaline C, D, and lantern batteries were once commonplace, but they aren’t very common anymore. I had a few NiMH (nickel-metal hydride) rechargeable batteries in AA form factor, who have the ambition to replace C and D batteries with a few adapter sleeves. I tested them and also found they could not sustain the required amperage under heavy draw.

I could also have multiple four-AA banks in parallel, or have separate power sources: one bank of four AA batteries just for steering servos, driving the remainder of the rover on some other power source. This complexity feels extremely inelegant and I can’t yet think of any reason why this path would be better than conceding that I will need a voltage regulator for steering servos.

So I moved the jumper to the other position, and started using a MP1584 buck converter breakout board set to produce 5.4V. This is between the valid range of 4.8V and 6V, and it is two volts under the nominal 7.4V of 2-cell LiPo I’m using to test this circuit. This two volt margin should be enough for MP1584 buck converter to work.

I used a MP1584 breakout board here because I had leftovers from a multipack (*) I bought for earlier projects, but I’m not confident they are the right device for the job. The datasheet claims it can sustain two amps of output with occasional spikes to maximum of 3 amps. Four SG90 micro servos would usually stay well under that limit, but their power consumption can spike occasionally making capacity planning unpredictable. At the very minimum I should put an electrolytic capacitor to buffer its output, and experimentation will tell me if I need more than that. I might also try to monitor the input voltage level.


(*) Disclosure: As an Amazon Associate I earn from qualifying purchases.

Power Distribution Complicates First Soldered Prototype Circuit

One of my ESP32 dev kit modules was too wide to be breadboard-friendly. But it had reduced pin count, making for a smaller overall footprint on a circuit board so I used it as the centerpiece for a perforated board prototype. This Micro Sawppy control board has the functionality of my breadboard prototype plus a few enhancements.

When I set out to plan my ESP32 control pin assignments, I laid them out roughly in the same physical relationship as they would be on a rover, hoping this would make control wire routing easier. I’m happy to report that this planning paid off, the motor control signal wires were very straightforward to route. For the servo control signals I reused wires from micro servos I negligently destroyed, and for DC motor control signal pairs I pulled solid core wires out of a CAT5E cable which was conveniently in already-twisted together pairs.

What I did not plan out was power distribution, which was something I knew was a challenge yet I had no plan beyond “maybe soldering would help”. Well, the lack of planning really showed in the utter mess of wiring that resulted, as more than half of the wires visible are for power distribution of one context or another. Which wasn’t helped by the fact I used thicker 22AWG wires for power distribution taking up far more space than low-amperage control signal wires. I definitely need to put some thought into power distribution for the next control board revision.

In the middle of the title image is a yellow jumper. When it is in place, the ESP32’s VIN pin is connected to this board’s battery power, letting it run standalone. This jumper must be removed before I upload new software to my ESP32. If not removed and the battery is not connected, the rover will try to run on power from the computer’s USB port which will not work possibly destroying the USB port. Or if the battery is connected, it will send battery voltage into the computer’s USB port which is also a bad thing. Unfortunately this setup doesn’t have the physical exclusion design in my solid-core wire breadboard version. I am scared of forgetting this jumper and want to bring physical exclusion back in a future revision.

I have another jumper on this soldered prototype board, but it serves a completely different purpose.

Another ESP32 Dev Kit Layout

After destroying a few SG90 micro servos and admitting I should have known better, I think the breadboard prototype circuit board has fulfilled its mission and it’s time to move on. My next iteration of Micro Sawppy rover control circuit will be a perforated prototype board with soldered connections. Having soldered connections will help give me more reliable connections and also reduce loss of electrical power from thin wires, which may or may not help with my power supply problems.

For my breadboard prototype I chose one of the ESP32 development modules that were narrow enough to fit well on a breadboard, leaving room on either side for jumper wires. (Left module on title image.) The narrow width meant there was no room left for pin labels, but I found a paper template that I could use to help with pin identification.

Now that I am building a soldered circuit, I no longer need to use the narrow module. So I pulled out a different development module I had bought(*) for exploring ESP32 development. (Right module on title image.) This one is wider and longer, giving us several advantages including screw mounting holes in each corner. Even thought it was longer, we have fewer pins to worry about, because this design didn’t bother to bring out the six pins corresponding to this ESP32 module’s built-in flash memory. We couldn’t use those pins in our design anyway so it was no huge loss. Conveniently, those flash memory pins were divided three on each side, so their removal still left the layout symmetric. A fourth pin was dropped from each side. On one side, an extraneous ground pin and on the other, pin zero was omitted. This is technically an available I/O pin but using it is tricky. It is a part of ESP32 startup process, the pin pulled to ground by the BOOT button. I had considered it a “pin of last resort” and avoided allocating it in my allocation scheme, so its subtraction from this module was fine for my use. All of the remaining IO pins are brought out on this module, and it even maintained the same relative ordering of those pins making it familiar to use. Helped by the fact that its more generous width also allowed onboard labels on these pins, eliminating the need for a paper template.

On board the module, I saw a very similar set of components. Two buttons “EN” and “BOOT” surround the micro USB port. There is a chip to handle USB serial translation, and an AMS1117-3.3 handles voltage regulation. I think I see a second LED which was absent from Espressif’s reference design but present in a few other dev kits I’ve used. But I also see one oddball: it appears a capacitor next to the EN button is at an unusual angle. When I see a surface mount component at a non-orthogonal angle, my first thought is a pick-and-place or soldering error. But this capacitor seems to be bridging two real pads and not dangling off into space, so it’s probably not an error. It might be some sort of a hack to address some problem discovered after the rest of the board was laid out, I’m not sure. As long as the module works I guess it really doesn’t matter. As a test I flashed my ESP32 test program and saw the second LED start blinking. Good enough to get to work.


(*) Disclosure: As an Amazon Associate I earn from qualifying purchases.