Solar Power

Roger Cheng

Initial Logic for Solar Monitor Project

I think I’ve got the hardware portions of my solar power monitor sensor node figured out, so I can write the first version of corresponding software logic. I have set out the following requirements:

  • Over-discharge protection: If battery voltage drops below a threshold, put the system to sleep to protect the battery.
  • Low solar output: When the solar panel isn’t generating any power, put the system to sleep.
  • Battery charging start: when panel power generation rises above a certain level for the first time that day, start charging the repurposed USB power bank battery.
  • Battery charging pause: If cloud cover causes a dip in solar power, pause charging.
  • Battery charging stop: Once battery cell voltage rises to a certain level, stop charging.
  • Sleep override: local hardware method to prevent deep sleep.

The ESPHome documentation for deep sleep described one way to prevent sleep using MQTT, keeping a node awake to receive firmware updates. But I wanted something even lower level hence the jumper and it became useful when I implemented the “low solar output, go to sleep” logic. Apparently INA219 component’s first power value always return zero. Which meant as soon as it booted up, that initial zero reading puts the node immediately back to sleep before it would even get on the network. (Never mind checking MQTT!) The solution is to switch from sampling values once a minute to sampling once a second, and make decisions based on average over a minute.

A different approach would be to go to sleep based on sun’s position in the sky, which can be queried and used in the Sun component. However, I expect this component has dependency on network connection (it needs to know the time, for starters) and would not be reliable if the network goes down. It also doesn’t know if the sun is obscured by clouds, so I think it’s better to use panel power output to decide what to do during the day. But I may explore using the Sun component in a future version to sleep all through the night instead of waking up every few minutes to fruitlessly check power level.

Strictly speaking, I don’t need to worry about stopping battery charging. I can supply 5V all day when panel delivers power, and trust the USB power bank charging circuit to keep the battery from being overcharged. But keeping lithium-ion cells full would shorten their useful life, so in the interest of battery longevity I’ll stop charging before full. On that topic: for optimal battery life I should charge it slowly over the course of the day, but I don’t have control over charging rate used by USB power bank.

One thing I don’t know yet is how the system will handle several rainy days in a row. I assume this panel can still generate enough power to charge an 18650 battery cell, but I might be wrong! I’ll have to wait for a long stretch of rain to come to Southern California, which may be a long wait. After seeing its behavior I can adjust for a future version.

Here’s version 1 of my ESPHome configuration YAML, and I expect to fine tune various hard-coded threshold values over the weeks ahead while I build more projects:

# Blue LED on the ESP8266 module signals connection status.
status_led:
  pin:
    number: 2
    inverted: true

# The goal is to charge once a day, and this flag tracks if we've already done it.
globals:
  - id: never_charged_today
    type: bool
    restore_value: no
    initial_value: "true"

# We can go to deep sleep to conserve battery, but sometimes we don't want to
# actually go to sleep. For example, when we need to upload a firmware update.
# Pin 13 is an input pin with internal pullup. It should be wired to a jumper
# that would ground the pin if jumper is present. Removing the jumper should
# disable going to deep sleep. To enforce this, call try_to_sleep script
# instead of calling deep_sleep.enter directly.
deep_sleep:
  id: deep_sleep_1

binary_sensor:
  - platform: gpio
    name: "Disable Sleep"
    id: sleep_jumper
    pin:
      number: 13
      mode:
        input: true
        pullup: true
    on_release:
      then:
        - logger.log: "Sleep jumper installed"
        - script.execute: try_to_sleep

script:
  - id: try_to_sleep
    then:
      if:
        condition:
          binary_sensor.is_on: sleep_jumper
        then:
          logger.log: "Sleep requested but staying awake due to override jumper"
        else:
          - logger.log: "Sleep requested and permitted by jumper"
          - delay: 5s # Allow sensor values to be sent.
          - deep_sleep.enter:
              id: deep_sleep_1
              sleep_duration: 10min

# This should be wired to a 1k resistor, which then connects to the enable pin
# of a power supply source. When ON, it should deliver power to charge the battery.
switch:
  - platform: gpio
    pin: D5
    id: charge_switch
    name: "Charge Battery"
    restore_mode: RESTORE_DEFAULT_OFF

# An I2C INA219 sensor monitors panel voltage, current, and calculates power.
i2c:
  sda: 4
  scl: 5

sensor:
  - platform: ina219
    address: 0x40
    shunt_resistance: 0.1 ohm
    max_voltage: 24.0V
    max_current: 3.2A
    update_interval: 1s
    current:
      name: "Panel Current"
      id: solar_panel_current
      accuracy_decimals: 5
      filters:
        sliding_window_moving_average:
          window_size: 90
          send_every: 60
          send_first_at: 15
    power:
      name: "Panel Power"
      id: solar_panel_power
      accuracy_decimals: 5
      filters:
        sliding_window_moving_average:
          window_size: 90
          send_every: 60
          send_first_at: 15
      on_value:
        then:
          # When power is low, put the board to sleep.
          # Note: upon boot, the first reading of current (and therefore power) always
          # seems to be zero, so we need to run moving average filters to ensure we
          # don't shut off immediately on power-up.
          if:
            condition:
              and:
                - sensor.in_range:
                    id: solar_panel_power
                    below: 0.01
                - sensor.in_range:
                    id: solar_panel_voltage
                    below: 3.0
            then:
              - logger.log: "Panel delivering low power, should go to sleep"
              - globals.set:
                  id: never_charged_today
                  value: "true"
              - script.execute: try_to_sleep
    bus_voltage:
      name: "Panel Voltage"
      id: solar_panel_voltage
      accuracy_decimals: 5
      filters:
        sliding_window_moving_average:
          window_size: 90
          send_every: 60
          send_first_at: 15
# ESP8266 ADC pin should be wired to a resistor just over 100kOhm to measure
# lithium-ion battery cell voltage. Values under calibrate_linear need to be
# customized for each board (and their resistors.)
  - platform: adc
    pin: A0
    name: "Battery Voltage"
    id: battery_voltage
    update_interval: 1s
    accuracy_decimals: 3
    filters:
      - calibrate_linear:
          - 0.84052 -> 3.492
          - 0.99707 -> 4.113
      - sliding_window_moving_average:
          window_size: 90
          send_every: 60
          send_first_at: 15
    on_value:
      then:
        - if:
            condition:
              and:
                - lambda: "return id(never_charged_today);"  
                - sensor.in_range:
                    id: solar_panel_power
                    above: 10
            then:
              - logger.log: "Panel has power, start charging for the day"
              - globals.set:
                  id: never_charged_today
                  value: "false"
              - switch.turn_on: charge_switch
        - if:
            condition:
              and:
                - switch.is_on: charge_switch
                - sensor.in_range:
                    id: solar_panel_power
                    below: 5
            then:
              - logger.log: "Charging paused due to low panel output"
              - globals.set:
                  id: never_charged_today
                  value: "true" # Resume charging if power returns
              - switch.turn_off: charge_switch
        # When battery is low enough to trigger this emergency measure, we
        # would not be able to activate charging ourselves. Charging needs to
        # be activated manually (or at least externally)
        - if:
            condition:
              sensor.in_range:
                id: battery_voltage
                below: 3.0
            then:
              - logger.log: "Battery critically low, should sleep to protect battery."
              - script.execute: try_to_sleep
    # We don't need a full charge to last through a day, so turn off charging
    # well before reaching maximum in order to improve battery longevity
    on_value_range:
      above: 4.0
      then:
        - logger.log: "Battery charge is sufficient"
        - switch.turn_off: charge_switch
Roger Cheng

Plotting Solar Panel Voltage and Power

Ever since I bought a cheap solar panel array from Harbor Freight, I’ve wanted to quantify its output. Most solar power instruments were designed for house-sized (or larger) arrays, overkill for my little panel. I had instruments from the world of remote-control hobbies to measure voltage and current, but they were only instantaneous values displayed on screen. I wanted more! This motivated my journey evaluating software tools like InfluxDB database, Grafana visualizer, Home Assistant, and ESPHome. Plus building on what I knew about hardware components like ESP8266 microcontroller, MP1584 buck converter, and INA219 sensor. All with the goal of building my own solar production monitoring system.

I had thought I could make the ESP8266+INA219 sensor node run exclusively on solar power, but I gave up on that and have built a temporary setup powered by old tired alkaline AA batteries. Enough to give me a graph!

This was a fairly sunny day, so cloud cover effects were minimal. The solar panel was connected to a MPPT charger. I could see the power levels (green line) climb smoothly from sunrise, reach a maximum at noon, then smoothly fading off to sunset. This was mostly as expected. The power curve is not symmetric because there are a few things blocking sunlight from the east, reducing morning power.

While the power curve was mostly expected, the voltage levels were not. Until this graph I didn’t know solar panels would jump up so high on voltage before producing meaningful amount of power. (I don’t know if the MPPT charger makes a difference here.) If I had known this, I probably wouldn’t have bothered trying to make a solar powered circuit turn on and off based on voltage. Or if I did, I would have tried anyway and knew why it failed. From this graph I learned that if I am to draw upon solar power, I need to base my decision on power and not on voltage. But in order to read power levels, I need power for my INA219! There’s probably a clever way to circumvent this Catch-22 without batteries, but I’m going to take the easy way out and incorporate a rechargeable battery.

Roger Cheng

Solar Startup Still Tricky

I have modified a second MP1584 buck converter module so that it would not activate until input voltage surpasses 13V, comfortably above the output voltage of 3.3V. I want to connect input pins to a solar panel so the associated components (ESP8266 WiFi microcontroller and INA219 voltage/current sensor) would be powered exclusively by the sun.

First is a test run with my bench power supply. Gradually increasing supply voltage starting from zero volts. Thanks to the modification, there were no odd behavior or sounds of a MP1584 trying to work with too low of an input voltage, which is great. As I increased voltage past the ~13V threshold, I saw the blue LED of my ESP8266 blink and it booted up as planned. This time, it was able to find INA219 on I2C bus, which is further than I got before.

Feeling optimistic, I connected this circuit to my solar panel at night and hoped I would wake up to find the system running. Sadly I woke up to disappointment, as there were no logged messages from the ESP8266. Probing the circuit with my volt meter, I confirmed a 3.3V supply voltage was present, but for whatever reason the ESP8266 failed to boot that morning. I manually disconnected and reconnected the circuit board, and this time ESP8266 booted up fine (now it has full daylight power) and started reporting values measured by INA219.

I don’t know what happened at sunrise. I hypothesize that when the solar panel output voltage rose past 13V, it has still yet to produce enough power to successfully start an ESP8266. So when MP1584 activated, it could supply 3.3V but not enough amperage to supply an ESP8266 through its boot process, putting it in a glitched state that was neither on nor off and stuck there until I power cycled the system. [UPDATE: Further experimentation found this hypothesis was correct, the panel would reach operating voltage well before generating appreciable power.]

I didn’t have my oscilloscope set up to capture the startup waveform to confirm or disprove this hypothesis. It’s clear there are additional subtleties I don’t know about starting up a circuit on solar power. Do I want to invest the time to learn and experiment with this problem domain? After thinking it over for a bit I decided “nah” and abandoned the idea of running everything exclusively on solar power. I’ll retreat to what I know and incorporate batteries into the system instead. Starting simple with household alkaline AA batteries.

Roger Cheng

Next Practice Round: MicroPython on ESP8266

I’ve been running through “Hello World” level microcontroller projects, using my little Harbor Freight solar array as a target subject. I just finished a trial run of programming an ESP32 using ESP-IDF platform and while analog input and power-saving deep sleep worked as expected, GPIO output pins did not. I thought I would use an external peripheral to get around the problem but, if I’m doing that, I might as well fall back to the cheaper ESP8266 used in a batch of Wemos D1 Mini clone(*) I had bought.

An earlier experiment with ESP8266 used the Arduino development platform. This time around, I’m trying MicroPython for the following reasons:

  1. With MicroPython I could modify my code without reflashing the entire ESP8266 as I had to do with Arduino.
  2. My Arduino sketch was a fairly short piece of code, but MicroPython promises to get going with even less code.
  3. Last time I looked at MicroPython I rejected it because it didn’t support a hardware peripheral I wanted to use. (MCPWM of ESP32.) This time I’m using just the ADC, which is supported.

I actually wanted to try CircuitPython, Adafruit’s derivative of MicroPython with changes focused on making things even easier for beginners to play with and something I first encountered on their HalloWing product. However, a part of CircuitPython is exposing the microcontroller as a USB mass storage device and letting us drag-and-drop Python source files onto the board for instant execution. This requires native USB support on the microcontroller, which is present on newer ESP32 variations but not the one I have. There used to be an ESP8266 version without the drag-and-drop but Adafruit stopped supporting that configuration some time ago.

Putting CircuitPython back on the “look at later” list, I changed focus to MicroPython. I brought up documentation for putting MicroPython on an ESP8266. In 15 minutes I had an interactive Python command line over serial port, though there were a few speed bumps.

(*) Disclosure: As an Amazon Associate I earn from qualifying purchases.

Roger Cheng

Switching to ESP32 For Next Exercise

After deciding to move data processing off of the microcontroller, it would make sense to repeat my exercise with an even cheaper microcontroller. But there aren’t a lot of WiFi-capable microcontrollers cheaper than an ESP8266. So I looked at the associated decision to communicate via MQTT instead, because removing requirement for an InfluxDB client library meant opening up potential for other development platforms.

I thought it’d be interesting to step up to ESP8266’s big brother, the ESP32. I could still develop with the Arduino platform with an ESP32 but for the sake of practice I’m switching to Espressif’s ESP-IDF platform. There isn’t an InfluxDB client library for ESP-IDF, but it does have a MQTT library.

ESP32 has more than one ADC channel, and they are more flexible than the single channel on board the ESP8266. However, that is not a motivate at the moment as I don’t have an immediate use for that advantage. I thought it might be interesting to measure current as well as voltage. Unfortunately given how noisy my amateur circuits have proven to be, I doubt I could build a circuit that can pick up the tiny voltage drop across a shunt resistor. Best to delegate that to a dedicated module designed by people who know what they are doing.

One reason I wanted to use an ESP32 is actually the development board. My Wemos D1 Mini clone board used a voltage regulator I could not identify, so I powered it with a separate MP1584EN buck converter module. In contrast, the ESP32 board I have on hand has a regulator clearly marked as an AMS1117. The datasheet for AMS1117 indicated a maximum input voltage of 15V. Since I’m powering my voltage monitor with a lead-acid battery array that has a maximum voltage of 14.4V, in theory I could connect it directly to the voltage input pin on this module.

In practice, connecting ~13V to this dev board gave me an audible pop, a visible spark, and a little cloud of smoke. Uh-oh. I disconnected power and took a closer look. The regulator now has a small crack in its case, surrounded by shiny plastic that had briefly turned liquid and re-solidified. I guess this particular regulator is not genuine AMS1117. It probably works fine converting 5V to 3.3V, but it definitely did not handle a maximum of 15V like real AMS1117 chips are expected to do.

Fortunately, ESP32 development boards are cheap, counterfeit regulators and all. So I chalked this up to lesson learned and pulled another board out of my stockpile. This time voltage regulation is handled by an external MP1584EN buck converter. I still want to play with an ESP32 for its digital output pins.

Roger Cheng

Initial Lessons on ESP8266 Arduino Sketch for InfluxDB

Dipping my toes in building a data monitoring system, I have an ESP8266 Arduino sketch that reads its analog input pin, converts it to original input voltage, and log that information to InfluxDB. Despite the simplicity of the sketch, I’ve already learned some very valuable lessons.

The Good

The Arduino libraries do a very good job of recovering from problems on their own, taking care of it so my Arduino sketch does not. The ESP8266 Arduino WiFi library can reconnect lost WiFi as long as I call WiFiMulti.run() periodically. And the InfluxDB library doesn’t need me to do anything special at all. I call InfluxDbClient.writePoint() whenever I want to write data. If the connection was lost since initial connection, it will be re-established and the data point written with no extra work on my part. I’ve had this sketch up and running as I’ve taken the InfluxDB docker container offline to upgrade to newer versions, or performed firmware updates WiFI access point which takes wireless offline for a few minutes. This sketch recovered and resume logging data, no sweat.

The Bad

ESP8266 ADC (analog-to-digital converter) is pretty darned noisy when asked to measure within a narrow range of its full range of voltages as I am. The full range is 0-22V, and I’m typically only within a narrow band between 12-14V. I tried taking multiple measurements and averaging them, but that didn’t seem to help very much.

This noisiness also made it hard to calibrate readings against voltage values as measured by my multi-meter. It’s not difficult to take a meter reading and calculation a conversion factor for an ADC reading taken at the same time. But if the ADC value can drift even as the actual voltage is held steady, the conversion factor is unreliable. Even worse, since the conversion is done in my Arduino sketch, every time I want to refine this value, I had to hook up a computer and re-upload my Arduino sketch.

Since I expect to add more data sources to this system, I also expected to query by source to see data returned by each. For this first iteration, I tagged points with the MAC address of my ESP8266. This is a pretty good guarantee of uniqueness, but it is not very human-friendly. Or at least not to me, as I’m not in the habit of memorizing MAC addresses of my devices.

The Ugly

As typical of Arduino sketches, this sketch is running loop() as fast as it could. Functionally speaking, this is fine. But it means the ESP8266 is always in a state of high power draw, with the WiFi stack always active and the CPU running as fast as it could. When the objective is merely to record measurements every minute or so, I could be far more energy efficient here.

Addressing these issues (and much more I’m sure) will be topic of future iterations. In the meantime, I have some data points and I want to graph them.

[Source code for this project is publicly accessible on GitHub]

Roger Cheng

Setting Up ESP8266 Arduino Sketch for InfluxDB

I think I’ve got the hardware side sorted out for building an ESP8266 that monitors voltage output of a solar panel, so it’s time to dive into the software side. I want to log this data into InfluxDB as a learning exercise, and the list of client libraries included a pointer to a GitHub repository for an InfluxDB client library for ESP8266 and ESP32 running on their Arduino core.

Arduino is not exactly the most fully featured development environment, so I’ve been doing my Arduino development using PlatformIO plugin of Visual Studio Code. However, this is the first time I’ve had to manually add a third-party library and it’s not the same as Arduino’s Library Manager so I had to go online for a little help like this forum thread. I learned I should go to https://registry.platformio.org/ and search for the desired library. Searching on “InfluxDB” resulted in several results, one of which is the same client library I found earlier but now with instructions on how to install into PlatformIO.

After compiling a simple test sketch, my serial output monitor returned gibberish. The key here is that baud rate must match between my platformio.ini configuration file:

monitor_speed = 115200

And my serial output initialization code in setup() function of my sketch:

  Serial.begin(115200);

Another configuration issue concern information necessary to connect to my home WiFi and my InfluxDB server. This little sketch needs that information to run, but I don’t want to include them in my source code since I intended to upload this to GitHub in a publicly accessible repository. I found a solution on StackOverflow: put my secret information in a separate secrets.h file. After I committed a basic version without any actual information, use the command git update-index --skip-worktree secrets.h to remove it from further Git activity. After that point, I could edit secrets.h and Git would not care to upload that information leaving my local secrets local, which is what I want.

Once all of these setup details were taken care of, I could dive into code and learn some valuable lessons out of the experience.

[Source code for this project is publicly accessible on GitHub]

Roger Cheng

Power Source Selection Jumper

I’m making a Wemos D1 Mini clone (with an ESP8266 at its heart) into a solar panel output voltage monitor. I plan to run it off the solar panel power as well, since it seems silly to involve another power source when it is already hooked up to one. However, having a buck converter supplying 3.3V to the ESP8266 means I need to avoid taking USB power at the same time. Having multiple voltage regulators on the same voltage plane is a bad thing. I don’t want to have dueling regulators when, for example, debugging over USB while it is connected to solar.

I know this is not a new problem, because every battery-powered USB device knows to switch between battery power and USB power. But I’m having trouble finding the right vocabulary to describe exactly my battery-less scenario. Using search terms like “isolating USB power” I usually find people who are trying to avoid ground loops for audio quality, or optical isolators for data, and other similar tasks which are useful but not the problem I’m trying to solve right now.

Momentarily stymied in my research, I switched over to devising my own manual solution. I’m routing the 3.3V output pin of my buck converter through a jumper on the circuit board. When the jumper is installed, the ESP8266 will run on the solar power it is measuring. When the jumper is removed, the module will run on USB power.

But I know myself, and I could not trust myself to remember to install/remove the jumper as the situation changes. Hence the next trick: placement of the jumper. I put it right next to the USB port so that the jumper could not be installed at the same time as the USB cable, ensuring that it is impossible to have both power sources active at the same time.

I think this mostly works, but I’m worried about the jumper pins. They are taller than I had expected and reach pretty close to the USB connector as we can see in this side view. When I plug/unplug the USB cable, I have to carefully avoid accidentally touching those pins. Accidentally shorting those pins would probably not damage the dev board, because electrically it is same as the jumper in place and at that point USB is not plugged in. However, touching the pin could connect voltage supply to ground and that might fry either the buck converter or something on my USB host, neither of which is ideal.

I didn’t like how accident-prone this design is, so I switched to plan B: cut the USB power line.

Roger Cheng

Setting Up ESP8266 Voltage Monitor

I’ve decided to start playing with an ESP8266 in a wireless data monitoring role. I plan to use it to measure power output of my little Harbor Freight solar array and log data into an InfluxDB database. The first draft will only deal with voltage, since I want to start with minimal hardware and the ESP8266 has only a single ADC channel.

In addition to the restriction of a single channel, the ESP8266 has an additional restriction that the voltage on this pin can only be from zero to one volt, a small sub range of its standard 3.3V signal. Since the analog in pin can’t be used for anything else, I learned many ESP8266 development boards went ahead and put a pair of voltage dividing resistors on board. Using my multi-meter I probed the lowest-bidder Wemos D1 Mini clone I purchased on Amazon (*) and found 220KΩ from the A0 pin on the dev board to ADC0 on the ESP8266 module then another 100KΩ to ground.

In a perfect world we would have 230KΩ and 100KΩ to divide 0-3.3V into 0-1V, but this is as close as we can get with commodity resistor values. Assuming going slightly over 1V would not damage the ESP8266, this just means we lose a little bit range if input voltage goes above 3.2V.

I will add to this, because I want to measure output of my solar panel, which has a much higher open circuit voltage. I’ve measured at a little over 20V. Reading online, the commonly quoted maximum voltage is 22V, which seems like a good target to design for. This means ideal additional resistance of 1880KΩ before connecting to A0 input pin. In the interest of keeping things simple with commodity resistor values, I’ll use two 1MΩ resistors in series and sacrifice a bit of resolution.

I want to power my sensor node from the same solar power it is measuring, which again can go up to >20V. The Wemos D1 Mini board has an onboard voltage regulator to take 5V USB power down to 3.3V required by the ESP8266. I can’t tell exactly what module it is, but I will assume it does not handle input voltage up to 22V. Instead I will pull another MP1584EN buck converter from the last batch I bought on Amazon.(*) I will configure it to output 3.3V and connect that to the 3.3V voltage plane on the Wemos D1 Mini.

Which leaves a problem: If I’m getting 3.3V supplied from the solar panel, it is important we do not connect to 5V USB power at the same time. Perhaps a switch is in order?

(*) Disclosure: As an Amazon Associate I earn from qualifying purchases.

Roger Cheng

Hello Wemos D1 Mini Clone

We can run the Arduino framework on an Espressif ESP8266 chip, but that was not its original purpose. It was originally designed as a WiFi bridge for small devices, handling wireless networking duties on behalf of another microcontroller. But as it turned out, the chip was more than capable to drive the whole show by itself for certain scenarios. Versions 1 and 2 of Pixelblaze, for example, ran on an ESP8266 even though my experience has been with the ESP32-based version 3. I haven’t had a project of my own where the ESP8266 made sense until the current InfluxDB logging project. It is time.

My electronics skill isn’t up to the level of directly using a bare ESP8266 chip. Using a module integrating ESP8266 chip with support circuitry (including a circuit board antenna) is close, but just beyond my comfort level. I went online looking for development boards for such modules and it appears many of them are clones of something called “Wemos D1 Mini“. A brief look at related web pages seem to indicate this rose to prominence alongside a development platform called NodeMCU. I don’t know which came first, the NodeMCU platform or the Wemos D1 Mini module, but they seem to be mentioned side by side in a majority of my search hits. Since I came into this project looking to use the Arduino core on an ESP8266 board, I’ll set aside the NodeMCU angle for now. I clicked “Buy” on a batch of Wemos D1 Mini clones from the lowest bidder of the day. (*)

When they arrived, I was pleasantly surprised to find that I had multiple options on connectors. The package came with three sets: I could have pins appropriate for a breadboard, or sockets appropriate for jumper wires, or passthrough connectors with both pins and sockets. And since none of them were pre-soldered on the board, I actually have a fourth option to “dead-bug” wire a circuit directly without any of those connectors. I think this will prove very useful to fit projects in tight spaces. Some people may gripe about having to heat up a soldering iron before this board is usable, and to them I shall point to this fishing line trick.

For my first ESP8266 project, I’ll stick with traditional pins, followed by mounting it in a perforated prototype board.

(*) Disclosure: As an Amazon Associate I earn from qualifying purchases.

Roger Cheng

Managed InfluxDB Arduino Client Access

While learning how to setup and run InfluxDB 2 in a docker container, I learned some administrative tasks require dropping to the command line. I had to do this by launching bash in a running instance via docker exec -it influxdb /bin/bash. I looked this up because creating less-privileged user accounts for non-administrative access had to be done in the command line, but it turns out InfluxDB manages access levels using authorization tokens and not user accounts. Creating a key with restricted permissions can be done within the UI and does not need the command line.

I wanted restricted access permissions for two main scenarios. The first ability is to add data to a specific table (sorry… um… measurement in InfluxDB terms) but unable to do other things like delete data. This is for my data sources, as I don’t want bugs in my data sources to corrupt or delete the database. The second ability is read-only access for analysis and displaying results. Again, I didn’t want a bug in a data dashboard to damage data.

Once I learned how to create these authorization keys scoped correctly to the minimum access necessary to fulfill a role, the next step is to use them. Looking over InfluxDB’s list of client libraries, I was fascinated to see Arduino listed among the expected list of programming languages. How does an ATMega328 talk to an InfluxDB instance? This got me really curious and clicked that link to a GitHub repository.

On the README.md I got my answer: this library is not for all Arduino boards, just the ESP8266 and ESP32 WiFi-enabled microcontrollers running their respective Arduino cores. Other Arduino platforms like Teensy or the original ATMega328 are excluded. I’ve played with the ESP32 for a bit and never had a real reason to look into its ESP8266 predecessor. This project will be the motivation to play with an ESP8266.

Roger Cheng

InfluxDB Investigation Skipping 1.x, Going Straight To 2.x

After I read enough to gain some working knowledge of how to use InfluxDB, it was time to get hands-on. And the first decision to make is: what version? The InfluxDB project made a major version breaking change not that long ago. And older projects like the Raspberry Pi Power Monitor project is still tied to the 1.x lineage of InfluxDB. Since this is an experiment, I would keep the footprint light by running InfluxDB in a Docker container. So when I looked on InfluxDB’s official Docker repository I was puzzled to see it had only 1.x releases. [Update: 2.x images are now available, more details below.] Looking around for an explanation, I saw the reason was because they did not yet have (1) feature parity and (2) a smooth automatic migration from 1.x to 2.x. This could mean bad things happening to people who periodically pull influxdb:latest from Docker Hub. While this problem was being worked on, InfluxDB 2 Docker images are hosted on quay.io/influxdb/influxdb instead of Docker hub.

I found it curious Docker didn’t have a standard mechanism to hold back people who are not ready to take the plunge for a major semantic version change, but I’m not diving into that rabbit hole right now. I have no dependency on legacy 1.x features, or a legacy 1.x database to migrate, or code using the old SQL-style query language. Therefore I decided to dive in to InfluxDB 2 with the knowledge I would also have to learn its new Flux query language that looks nothing like SQL.

Referencing the document Running InfluxDB 2.0 and Telegraf Using Docker, I quickly got a bare-bones instance of InfluxDB 2 up and running. I didn’t even bother trying to persist data on the first run: it was just to verify that the binaries would execute, and that the network ports were set up correctly so I could get into the administration UI to poke around. On the second run I mapped a volume to /root/.influxdbv2 so my data would live on after the container itself is stopped. Now all I need to do is to get some data to store!

[Update: After InfluxDB Docker Hub was updated to release version 2 binaries, the mapped volume path changed from /root/.influxdbv2 to /var/lib/influxdb2. See the InfluxDB Docker Hub repository for details under the section titled: Upgrading from quay.io-hosted InfluxDB 2.x image. In my case it wasn’t quite as straightforward. The migration was introduced in InfluxDB 2.0.4, but I got all the way up to 2.1.1 from quay.io before I performed this migration. A direct switch to 2.1.1 did not work: it acted as if I had a new InfluxDB instance and didn’t have any of my existing data. Trying to run 2.0.4 would fail with a “migration specification not found” error due to a database schema change introduced in 2.1.0. Fortunately, running 2.1.0 docker image appeared to do the trick, loading up all the existing data. After that, I could run 2.1.1 and still keep all my data.]

Docker compose file I used to run 2.1.1 image hosted on quay.io, data stored in the subdirectory “mapped” which is in the same directory as the docker-compose.yml file:

version: "3.8"

services:
  server:
    image: quay.io/influxdb/influxdb:2.1.1
    restart: unless-stopped
    ports:
      - 8086:8086
    volumes:
      - ./mapped/:/root/.influxdbv2/

Update: here’s the version to use latest Docker hub image instead:

version: "3.8"

services:
  server:
    image: influxdb:latest
    restart: unless-stopped
    ports:
      - 8086:8086
    volumes:
      - ./mapped/:/var/lib/influxdb2/
Roger Cheng

Learning InfluxDB Basics

I’ve decided to learn InfluxDB in a project using it to track some statistics about my little toy solar array. And despite looking at some super-overkill high end solutions, I have to admit that even InfluxDB is more high-end than I would strictly need for a little toy project. It will probably be fine with MySQL, but the learning is the point and the InfluxDB key concepts document was what I needed to start.

It was harder than I had expected to get to that “key concept” document. When someone visits the InfluxDB web site, the big “Get InfluxDB” button highlighted on the home page sends me to their “InfluxDB Cloud” service. Clicking “Developers” and the section titled “InfluxDB fundamentals” is a quick-start guide to… InfluxDB Cloud. They really want me to use their cloud service! I don’t particularly object to a business trying to make money, but their eagerness has overshadowed an actual basic getting started guide. Putting me on their cloud service doesn’t do me any good if I don’t know the basics of InfluxDB!

I know some relational database basics, but because of its time-series focus, InfluxDB has slightly different concepts described using slightly different terminology. Here are the key points, paraphrased from the “key concepts” document, that I used to make the mental transition.

  • Bucket: An InfluxDB instance can host multiple buckets, each of which are independent of the others. I think of these as multiple databases hosted on the same database server.
  • Measurement: This is the one that confused me the most. When I see “measurement” I think of a single quantified piece of data, but in InfluxDB parlance that is called a “Point”. In InfluxDB a “measurement” is a collection of related data. “A measurement acts as a container for tags, fields, and timestamps.” In my mind an InfluxDB “Measurement” is roughly analogous to a table in SQL.
  • Timestamp: Time is obviously very important in a time-series database, and every “Point” has a timestamp. All query operations will typically be time-related in some way, or else why are we bothering with a time-series database? I don’t think timestamps are required to be unique, but since they form the foundation for all queries, they have become de-facto primary keys.
  • Tags: This is where we start venturing further away from SQL. InfluxDB tags are part of a Point and this information is indexed. When we query for data within a time range, we specify the subset of data we want using tags. But even though tags are used in queries for identification, tags are not required to be unique. In fact, there shouldn’t be too many different unique tag values, because that degrades InfluxDB performance. This is a problem called “high series cardinality” and it gets a lot of talk whenever InfluxDB performance is a topic. As I understand it, it is a sign the database layout design was not aligned with strengths of InfluxDB.
  • Fields: Unlike Tags, Fields are not indexed. I think of these as the actual “data” in a Point. This is the data we want to retrieve when we make an InfluxDB query.

Given the above understanding, I’m aiming in the direction of:

  • One bucket for a single measurement.
  • One measurement for all my points.
  • “At 10AM today, solar panel output was 21.4 Watts” is a single point. It has a timestamp of 10AM, it is tagged with “panel output” and a field of “21.4 Watts”

With that goal in mind, I’m ready to fire up my own instance of InfluxDB. But first I have to decide which version to run.

Roger Cheng

Investigating Time Series Data

I’ve had a little solar power system running for a while, based on an inexpensive Harbor Freight photovoltaic array. It’s nowhere near big enough to run a house, but I can do smaller things like charge my cell phone on solar power. It’s for novelty more than anything else. Now I turn my attention to the array again, because I want to start playing with time series data and this little solar array is a good test subject.

The motivation came from reading about a home energy monitoring project on Hackaday. I don’t intend to deploy that specific project, though. The first reason is that particular project isn’t a perfect fit for my little toy solar array, but the real reason is that I wanted to learn the underlying technologies hands-on. Specifically, InfluxDB for storing time-series data and Grafana to graph said data for visualization.

Before I go down the same path, I thought I would do a little reading. Here’s an informative StackExchange thread on storing large time series data. InfluxDB was one of the suggestions and I’m sure the right database for the project would depend on specific requirements. Other tools in the path would affect throughput as well. I expect to use Node-RED for some intermediary processing, and that would introduce bottlenecks of its own. I don’t expect to be anywhere near hundreds of sampling points per second, though, so I should be fine to start.

The original poster of that thread ended up going with HDF5. This is something developed & supported by people who work with supercomputer applications, and again had very different performance requirements than what I need right now. HDF5 came up in a discussion on a revamped ROS logging format (rosbag) and it linked to some people unhappy with HDF5. So there are definitely upsides and downsides to that format. Speaking of ROS2… I don’t know where they eventually landed on the rosbag format. Here is an old design spec absent from the latest main branch, but I don’t know what eventually happened to that information.

While I’m on the subject, out of curiosity I went to look up what CERN scientists use for their data. This led me to something called ROOT, a system with its own file format. Astronomers are another group of scientists who need to track a lot of data, and ASDF exists to serve their needs.

It was fun to poke around at these high-end worlds, but it would be sheer overkill to use tools that support petabytes of data with high detail and precision. I’m pretty sure InfluxDB will be fine for my project, so I will get up to speed on it.

Roger Cheng

Functional and Useful 100W Solar Array

Once the Monoprice PowerCache 220 was connected to the Harbor Freight 100W Solar Kit (Item #63585), we have everything we need to gather a little bit of sun power and make use of it every day. Given the non-optimal solar panel position and the fact we’re close to the winter solstice, this is just about the worst case scenario for solar power. Nevertheless this system has been gathering enough power to keep all the battery-powered electronics in the house charged up. This includes daily use & charge items like cell phones, tablets, and laptop computers. Plus the occasional items like a digital camera.

There is much more we can do to improve performance of this system, but it has met a minimum level of satisfactory performance so we can leave it running as-is for a while and switch gears to other projects. The focus will eventually return to this solar power system and here are two candidate projects for later:

Physical tracking: the panels are currently just sitting indoors vertically set against a south-facing window. It was done as an easy nondestructive way to experiment. When it comes time to improve upon this configuration, we can build a more permanent outdoor installation that angles into the sun. Maybe even motorized sun tracking throughout the day!

Electrical tracking: at the moment, the solar panel output voltage is dictated by the battery being charged. This is convenient and simple to implement but not the most efficient. We can buy (or design and build our own) a “maximum power point tracking” (MPPT) charger that keeps the solar panel voltage at its most efficient level and transform that to the correct battery charging voltage. It costs some power to do this tracking & voltage conversion, but if implemented correctly, the additional power will more than offset the cost.

We’ll add these projects to the bottom of the “to-do” list. For now, behold the glory of electronics being charged by sun power.

PowerCache 220 At Work cropped

Roger Cheng

Solar Charging Plug for Monoprice PowerCache 220

The product description page for Monoprice PowerCache 220 (Item #15278) had pictures showing its solar charging input as two round ports, but there were no connector specifications. Since it was advertised as a solar power cache, it would make sense for them to be MC4 connectors. This turned out not to be the case – they were actually two identical sockets that are electrically wired in parallel.

The device comes with an AC adapter for the charging port, but using household power would miss the point of the exercise. Neither do we want to lose the option to charge from grid power, so we don’t want to just cut off the charging plug. We should find an identical plug for solar charging.

The calipers indicate this connector has an outside diameter of 7.9mm and an inside diameter of 5.5mm. There is a small pin in the center roughly 0.5 mm in diameter. Putting those dimensions into search returned a few candidates and a large number of Lenovo laptop power adapters. A fortunate association, as there is a Kensington “Universal Laptop Power Supply” already on hand. It came with a set of interchangeable plugs to fit different laptop brands. This one was purchased to power a Dell, so the Lenovo plug has sat unused. But not for long!

Kensington Laptop Power

The Lenovo laptop plug matches all the diameters, and is slightly longer. It appeared to fit in the socket nicely. Since the two charging ports are electrically parallel, we could plug the AC adapter into the other charging port. This allowed us to read the voltage on the pins of the Lenovo plug so we know where to solder the positive and negative wire.

The plastic housing of the Lenovo plug was damaged in the soldering process so this plug will probably never fit on the Kensington laptop adapter again, but that’s fine. It now has a new purpose as solar charging plug for Monoprice PowerCache 220.

Lenovo now Solar plug

Roger Cheng

Hunt for AC Inverter Finds Monoprice PowerCache 220

Now that we have a 20 amp-hour 12 volt battery, charged by a solar array delivering up to 125 watt-hours daily. To put that power to use, we’ll need an inverter to convert battery power into household 120 volt AC power. This way the collected solar power can be used for more than just charging USB devices.

An old inverter was dug out of the old equipment collection, but it could only sustain about a minute of work before it would stop, reset, and restart. This isn’t great for the electronics, but what made it intolerable to humans were the cascade of noises that devices emit when charging began. Hearing the symphony roughly once a minute was unacceptable so the search begins for a replacement AC inverter.

The Harbor Freight lineup of AC inverters were obvious candidates. Starting from a basic model just under $20 and going up from there. While investigating options outside of Harbor Freight, one stood out: Monoprice #15278  “PowerCache 220”

It is designed exactly for the task we’re building for. It can accept power from a solar array to charge its 18 amp-hour 12 volt battery. That power can be consumed directly as 12 volts DC, as 5 volt USB power, or as 120 volt AC power.

The PowerCache mostly duplicates the components already in the current solar experiment setup. Buying one might be called wasteful, but for the sake of exploration we’ll call it redundancy. This nearly doubles the battery capacity and allows more ways to put solar power to use. It is also more user-friendly than the current maze of wires and connectors. It is an enclosed unit therefore easily portable. This might come in handy if we ever have a reason to take a little portable power source on the go.

So the search that began as a search for a simple AC inverter ended up with purchase of an integrated unit that included the AC inverter and basically everything else short of the solar panels themselves.

PowerCache 220

Roger Cheng

Initial Results of Solar Generation by 100 Watt Kit

Once the replacement battery arrived, it was possible to use the power captured by the Harbor Freight 100 watt solar kit. (Item #63585) The E-Flite Power Meter tracks the cumulative solar energy pumped back into the battery every day. Over several sunny days with minimal cloud cover, the daily tally ranged from 8 amp-hours to nearly 10 amp-hours. On an overcast day, the daily tally struggled to reach 4 amp-hours. (Reminder: the solar panels are not optimally placed to face the sun in these experiments.)

At the battery voltage range of 12 to 13 volts, this means a sunny day gives us about 125 watt-hours of electricity. (12.5 volt * 10 amp-hour) An overcast day’s output drops to about 50 watt-hours. (12.5 * 4) Since the sunniest and most productive times for solar largely overlaps with the most expensive times of the time-of-use electricity rates, we can try our best to make this solar array look good, by comparing against the highest rate of 35 cents per kilowatt-hour.

Even when using that expensive rate, a sunny day’s generation only works out to a tiny bit over 4 cents of grid electricity. A cloudy day couldn’t quite make up to 2 cents worth. Rough estimates point to a meager 10 dollars a year of savings on the electric bill.

Fortunately, we’re not doing this for money, and there is room for improvement as well. The solar array can be better aligned with the sun which, from earlier experiments, we know will make a huge difference. But a more immediate concern is the fact only a few items around the house can directly use DC power. There aren’t enough cell phones and tablets to consume 125 watt-hours a day.

In order to make solar power more useful, we’ll need an inverter to take the battery’s 12 volt DC power and turn it into household 120 volt AC.

20Ah Battery.jpg

Roger Cheng

Initial Use of 100 Watt Solar Kit Hampered By Battery

For the initial round of testing, the solar panels of the Harbor Freight kit (item #63585) was set up in a very temporary way: they were leaned against the south-facing windows of the house. To measure the output, we’re enlisting another member of the parts pile, unearthed when we were digging for the lead-acid battery: an E-Flite Power Meter designed to measure electric consumption of remote-controlled aircraft motors. It can handle the expected range of voltage and amperage and as a bonus it also tracks the total power in milliamp-hours.

Based on experiments with the small 1.5 watt panel, we knew not to expect the advertised 100 watt output with this sub-optimal, non sun-tracking orientation. The power meter gave confirmation: over the sunlight hours of a winter day, the panel generated power ranging from 20 to 30 watts. This is roughly in line with the small 1.5 watt panel experiment indicating sub-optimal placement returned as little as 25% of the power compared to directly facing the sun.

The solar charge controller allowed the battery voltage to rise to 14.4 to top it off, then disconnected the battery from the panel to avoid over-charging. Once the voltage dropped to 13.8 volts the controller kept the battery at this sustained charge level for as long as the solar panel could keep it there. All this fits expectation of a charge controller doing its job properly.

But something was wrong when withdrawing power from this assembly after the sun went down. Trying to charge a cell phone at night, battery voltage quickly dropped below cutoff threshold of 12 volts and the controller halted operations. There seems to be usable battery capacity remaining but the battery should have been able to hold roughly 12.5 to 13 volts for the majority of the power delivery period.

Looks like this battery did suffer some damage when it dropped down to 6 volts while sitting neglected in storage. Time to head over to Amazon and buy a replacement lead-acid battery with a good amp-hour per dollar ratio. The best ratio varies from day to day pricing fluctuation but at the moment meant this 20 Ah unit. (Disclosure: As an Amazon Associate I earn from qualifying purchases.)

Roger Cheng

Harbor Freight #63585 100 Watt Solar Kit

Seeking more power than what a 1.5 watt solar panel could provide, it’s time to step up to the 100 watt solar kit, Harbor Freight item #63585. The manual, posted online as a PDF, fails to describe a few useful details which we’ll cover here.

Every product picture showed the four panels lined up in a row. But in fact the four panels are capable of standing separately as each panel is in their own frame and has their own folding stand. Bolting them together is optional. If the panels are to be deployed and stowed frequently, leaving them separate might make sense as the panels are much easier to handle individually.

The package content lists wires but not their length. Each panel has a 3 meter long wire permanently attached. This wire terminates in a connector common to Harbor Freight solar products but its exact type specification is unknown. It is definitely not the MC4 connector common in rooftop solar installations.

(UPDATE: Thanks to a tip in the comments, we now know this is a connector commonly used in the automotive world and can be purchased from auto parts stores. For example it is commonly used to make electric connections to trailers. While this connector follows the pattern of SAE J928 and J1239, it is not explicitly covered by either specification.)

The four panels connect into a 4-to-1 module. The four wire side are half a meter long, and the unified side has a 3 meter long wire towards the controller. A final half-meter long adapter has the unknown HF solar automotive connector on one end and a barrel connector on the other. (~5.5mm OD, ~1.5mm ID, 12mm length) The barrel connector fits into a corresponding jack on the controller.

HF 63585 100W power adapter

Adding it all up: Each of the panels can be up to 3.5 meters from the central 4-to-1 hub, and that hub can be up to 3.5 meters from the controller. The package includes a 1 meter cable to connect controller to battery.

The kit included two LED light bulbs, each of which have a 5 meter long wire. Curiously, the long wire ends in a standard light bulb socket. But instead of the 120V AC household voltage we would expect from such a socket, it carries the battery DC voltage. This is a decidedly nonstandard and confusing way to do things. (UPDATE: An earlier version of this paragraph incorrectly stated 120V AC conversion took place, a bad assumption based on the standard light bulb socket. Voltage meter told the truth and paragraph has been rewritten.)

HF 63585 100W LED bulbs

The simple charge controller covers the basics, guarding against battery overcharging and over-discharging at adjustable voltage thresholds. The manual claims there is over-current protection as well, but there appears to be no way to adjust the current limit, either for charging or for discharging.