Author: Roger Cheng

Roger Cheng

Budibase Documentation: Mind the Gap

I want to learn Budibase and I hope it will be a tool to solve my “Excel is not a database” situation with my personal finance tracking. Keeping that project in mind to anchor my study, I returned to Budibase documentation ready to learn more. They had an excellent “Overview” page, followed by an informative “Quickstart” project. Given those two precedents, I had expected “Quickstart” to be followed by a smooth ramp-up for beginners. Turns out I was overly optimistic. I found a significant gap between “Quickstart” and the knowledge needed to build my own Budibase app.

Going down the documentation index sidebar, “Quickstart” is followed by “Guides and resources”. This is a collection of solutions to common problems, sometimes called a recipe cookbook. I’m sure it’s a great resource, but for a beginner like myself coming right out of “Quickstart”, a cookbook is not what we need. Beginners lack experience to understand the solutions being offered. Even worse, we don’t even necessarily understand the problems being solved. I suffered a similar problem when I encountered the Node-RED cookbook.

After the “Getting Started” section, Budibase documentation had sections titled “Data”, “Design”, and “Bindings” but I saw nothing telling a beginner how they tied together. I was undaunted — I’ve been in similar situations before — so I dove in to read those sections and learn what I can. There were a lot of cross-referencing in those sections (each had references to the other two) meaning it wouldn’t help to read the three sections in any particular order. I had to accept I wouldn’t understand everything the first pass. Then, after reading all three sections once, return to read them all a second time. Only then did I feel I have a basic grasp of how a Budibase app worked.

I think there’s room for improvement here. If my experience is representative, I could see potential Budibase users getting discouraged trying to jump over this post-“Quickstart” gap. In hindsight, I think a few sentences at the end of the “Quickstart” guide would have gone a long way to help. Here’s my first draft:

Budibase Data Flow

This tutorial covered the core of every Budibase app, but things will be done differently beyond a tutorial.

  • Obviously real Budibase apps will not use our sample data. Visit Introduction to data to see how Budibase can work with your data.
  • This tutorial used autogenerated screens to get started quickly with a generic table interface. A good application will tailor an interface to best display and interact with its underlying data. Data in design goes into more details on how to do so.
  • Our autogenerated tables also handled linking data between database and user interface elements. These links are called bindings and Introduction to bindings covers how to define them.

I might submit this, or an evolution of it, as a proposed contribution after I get some Budibase experience to be confident this is actually correct. There’s a lot to learn.

Roger Cheng

Resurrecting Project Idea: Personal Finance Software

I took a quick look under the hood of Budibase to get a rough idea of its capabilities. Before I start reading Budibase documentation in detail, I also want to designate a “north star” project. I’ve found this helps organize my thoughts when I explore a new piece of technology. Without a goal, I risk wandering aimlessly and losing interest. Keeping a project idea in mind helps me stay focused and absorb information in terms of how it relates to my project. For Budibase, I decided to dust off my idea to build personal finance software tailored exactly for myself.

I’ve used personal finance software to track my money ever since I had money to track. I started with Microsoft Money. After Microsoft shut it down, I switched to Intuit Quicken. As the world became more online, I switched to Mint. It didn’t take long before I decided I didn’t like doing my personal finances that way. (Well before Intuit bought and eventually shut down its competitor.)

When I started with Money, the usage pattern is for me to enter my financial transactions into my PC software. Then when I received my bank statements, reconcile what I entered with my statement to make sure everything matched up. By the time of Mint, such usage was still possible but no longer the focus. I was encouraged to connect to my financial services online to download account activity. Everything is entered and reconciled automatically. I totally understand why this was done. Most people don’t want to spend the time to manually enter their transactions, then manually reconcile them. The modern usage pattern takes no time. But to my mind, such automation took away the first layer of awareness of my own finances. It’s easy for small fraudulent transactions to slip through unnoticed, for example. I didn’t like it.

After transactions are entered and reconciled, all personal finance software include analysis tools to help the user better understand where their money has gone. But as a data nerd, such graphs and charts were never good enough. I can bring up a chart, click to see more details about what contributed to that graph, and the drill-down always ends before I am satisfied with the level of detail. I understand that most people wouldn’t care to spend time for such analysis, but I do! Spending reports. Flow of money. Account activity. More, I crave more!

I have come to accept the fact my own desires are not aligned with the general populace. Which meant I will have to roll up my sleeves and do it myself. Microsoft Excel seemed like an obvious choice as spreadsheets are intrinsically linked to finance. I started a spreadsheet to track my expenses and things worked well enough at the beginning. As my records grew, though, I started hitting limits of my Excel skills. It was good motivation for me to learn more Excel as I go, but eventually I recognized my mistake. Yes, Excel is the king of finance, but I’m not doing any financial calculations. (No compound interest, etc.) What I’m really doing is record keeping, which is a database job! I can continue learning Excel techniques to compensate for the fact it isn’t a database, or I can switch to using a database.

I’ve approached this project idea many times. I had this in the back of my mind when I took Codecademy’s PostgreSQL and MongoDB database classes, for example. I even got as far as creating a GitHub repository called “rogmoney”, but I didn’t do anything beyond setting up a Node.js environment. Looking at the timestamp I believe my thought was to build a personal finance app on Angular but I never even got as far as committing an empty Angular boilerplate app. Now I will make another attempt. It will help me learn Budibase whether the attempt results in my ideal personal finance tool or not.

Roger Cheng

Quick Survey of Budibase Foundation

I’m playing with Budibase on a self-hosted instance courtesy of Docker Compose. I didn’t edit the docker-compose.yaml file provided by Budibase but I did open it up to look around to see what Budibase was built atop of. Later I found answers within Budibase documentation which included a page on its system architecture and other scattered information on how it was built. Most of these building blocks are things I might use in my own future projects, with one major exception.

CouchDB

I knew CouchDB is one of the “NoSQL” databases out there, but not much more than that. My own NoSQL database adventure was with MongoDB and I don’t yet know enough to tell when I might use one versus another. Budibase uses CouchDB for its own internal operational data like user accounts. It is also available as the default store for client app data if I don’t want to connect to an external database. My to-do list includes better understanding the tradeoffs between those two paths.

Redis

Not everything involved in running an application needs to be written to disk. Sometimes an in-memory data storage system like Redis is the right tool for the job, because it is far faster than any disk. I’ve known Redis existed but never had a project that needed its capabilities. Budibase says it uses Redis as a high performance cache store.

MinIO

One thing that caught my eye about Budibase is file-handling ability. Meaning data entries aren’t limited to classic database types like dates or numbers or text, I can also upload file attachments. I had guessed it was implemented using something like PostgreSQL binary data format, but I was wrong. Attachments are actually stored in an instance of MinIO which I knew nothing about until now. A quick skim of documentation looks like MinIO is a way to get much of the functionality of AWS S3 but on my own data storage instead of AWS datacenter hardware. Like CouchDB, MinIO also used to store files for Budibase internal operation in addition to storing client app data.

Svelte

I was curious how Budibase built the HTML UI I had been looking at, and part of the answer is Svelte. This was on my “maybe look into later” list due to its connection with Vite, which I briefly looked at earlier. It’s all layers upon layers, and there appears to be yet another layer on top: SvelteKit. I didn’t see any mention of SvelteKit, though, so I think Budibase is using things at Svelte level.

Adobe Spectrum

Another component of Budibase HTML UI is Adobe’s Spectrum design system. I don’t recall ever coming across Spectrum before, but in hindsight it made sense Adobe would create its own interface design. Several years ago I became infatuated with Google’s Material Design system, and it was a big motivation for me to learn Google’s Angular platform. Since then Google corporate behavior turned me from a fan into a skeptic, and I’m not inclined to further pursue either Angular or Material Design. Adobe Spectrum Design would be an alternative… if it weren’t for the fact I’m even less fond of Adobe business practices. The fact Budibase uses Adobe Spectrum is not a deal-breaker against Budibase, but I’m never going to use Spectrum for my own web development projects.

Speaking of my own projects, I need to pick something interesting I can use to learn Budibase as I try to turn idea into reality. Fortunately I have no shortage of project ideas and have a long standing “Excel is not a database” problem I’ve wanted to solve: personal finance tracking.

Roger Cheng

Self-Hosting Budibase (Docker Compose)

I went through Budibase quick start guide and it made a great first impression. I want to learn it and add it to my toolbox for times when I need to build a data web app in a hurry. The quick start used Budibase cloud hosting for an easy low-friction way to kick the tires. Good enough for me to continue, but before I do it was important that I continue with an instance of Budibase I run for my own use. Wary of the example set by AWS Honeycode, which Amazon had shut down and took all user data offline with it, it was critically important that I can run my own local instance on data I control. This is a pretty common requirement so Budibase offers many alternatives to their cloud-hosted solution. I clicked on the link for deploying via Docker Compose and it turned out to be really easy.

I didn’t expect it to be that easy. I was no stranger to install procedures that went awry and causing collateral damage elsewhere. I spun up a Proxmox virtual machine running Ubuntu server 24.04 LTS for the exclusive purpose of experimenting with Budibase. If anything goes wrong in a VM, damage should be contained and I can easily erase everything and start over. I followed Budibase instructions to install Docker Compose then downloaded their docker-compose.yaml file for deploying Budibase and an .env file for configuration variables. I didn’t edit their docker-compose.yaml file, just the .env file to set my own passwords. I then typed “docker compose up“, waited for all the startup and deployment procedures to run, then pointed my browser at the address for my Budibase virtual machine. I was greeted with the initial setup screen to create an account on my newly-created Budibase instance. No mysterious failures, no inscrutable error messages, excellent.

I repeated the quick start tutorial on my own instance, and it worked exactly as it did on the cloud hosted instance. For the most part this is good, but not always. My first rude surprise was finding that Budibase app backup was locked off. Backup was clearly indicated as a Premium subscription tier feature, but I had thought that applied only to the cloud-hosted service. I was wrong: subscription features are locked away whether I’m running on their cloud hosting platform or on my own. If Budibase is truly and fully open source I suppose I can clone their GitHub repository and build my own version without these locks. Even if I don’t want to do that, the fact I’m running with Docker Compose means I can back up my data via Docker volumes. So, yes, there are solutions, but this is proof that levers still exist for Budibase to restrict self-hosted instances.

This is a demerit against Budibase, but not a deal-breaker. After all, I’m putting up with much the same risk in other self-hosted services like Plex Media Server, though in that case I have an alternative Jellyfin lined up. For now it’ll just be a note here on my project notebook and I will continue exploring Budibase.

Roger Cheng

Budibase Quickstart Is A Great Tour

I want to investigate tools for quickly building data applications and thought Budibase was the most promising candidate. Since the main pitch is ease of getting something useful up and running, they have put together a quick start guide to serve as a lightning tour of product features. Their effort to make a great first impression certainly worked on me.

Minimizing barriers to entry, we are pointed to their cloud-hosted service where we register to create our quick start app without worrying about setup or deployment. Then a sample database is only one click away. I examined the sample database and found a decently structured example with multiple tables and relationships between them. I had expected to see an easy single table database and was impressed they gave us something more representative of real world applications.

Once sample database tables were in place, the tour takes us to the interface design screen. One of the default templates is an editable table that would be comfortably familiar for any spreadsheet user. Simple data navigation like sorting by a column or searching in a column works much the same way as well, except this editable table also has a basic ability to interact with relational database tables.

If the goal is to take a small first step away from abusing Excel as a database, Budibase lives up to its promise. It really did take only a few minutes to set up a web app that presents an Excel-like interface but with underlying relational database capabilities managed with web app access control and data validation capabilities. It’s a credible “Minimal Viable Product” and if that’s all somebody needs, they can stop right there and have a perfectly usable tool.

But of course that is only the beginning. The real power comes from building interfaces better tailored to the task at hand instead of a spreadsheet-like table. For data entry, Budibase offers the usual set of form elements (text entry, number entry) plus a few intriguing surprises (file attachment upload?) Playing around with these tools I got the distinct feeling this section of Budibase is aiming to surpass Google Forms for ease of use. [UPDATE: Yep, there’s a page in Budibase documentation dedicated to pitching itself against Google Forms.] I’m not enough of a Google Forms user to judge if Budibase has succeeded, but I see no reason why I need to go back to Google Forms.

This quick start guide ends with links to other sections in Budibase documentation for further exploration. I like what I see so far, and I want to continue exploring, but on my own self-hosted instance instead of their cloud-hosted one.

Roger Cheng

Window Shopping Budibase

Occasionally I’ve had project ideas that require a database to keep track of everything, and then I would shelve the idea because I didn’t want to spend the time. Or even worse, try to do it with a spreadsheet and suffer all the ills of a square peg in a round hole. I want an easier way to use databases. The good news is that I’m not the only one with this problem and many solutions exist under the marketing umbrella of “low-code” or “no-code” web app platforms. Skimming available information, I think Budibase is worth a closer look.

Budibase is one of the “low-code” platforms and I understand this to mean I don’t have to write any code if my project stays within the lane of a supported usage pattern. But if I must go outside those boundaries, I have the option of inserting my own code to customize Budibase instead of being left on my own to re-implement everything from scratch. Sounds good.

On the business side, Budibase is a free open source project with source code hosted on GitHub. Compiled binaries for self-managed deployments are available a few different ways, including Docker containers which is how I’d test things out. Users are allowed to run their own apps on cloud infrastructure of their choice. However, the licensing agreement forbids reselling Budibase itself as a cloud-hosted service. That privilege is reserved exclusively to Budibase themselves and serves as an income stream alongside consultancy contracts to build apps for businesses. I can support this arrangement as a tradeoff between proprietary commercial software and totally free open-source projects without a reliable income stream. This way Budibase is less likely to shut down and, even if it does, I should be able to limp along on a self-hosted setup until I can find a replacement.

Which brings me to the final point: the database. For the sake of simplicity, some of these low-code/no-code app platforms are built around their own database and assume full control over that database from beginning to end. In some cases this puts my data into a black box I might not be able to port to another platform. In contrast, I can build a Budibase interface on top of an existing database. And that database will still be available independently if Budibase goes away, or if I just want to explore something else. I like having such options and the data security it implies.

I like what I see so far, more than good enough for me to dive into Budibase documentation. Learn how I can translate its theoretical benefits to reality. Starting with an excellent quick start guide.

Roger Cheng

Many Options For Create/Read/Update/Delete (CRUD) Web Applications

In my quest to stop abusing Excel for database tasks, I returned to the world of web development. I remember studying web development for a while before I came across the term “CRUD app”. Though I might have encountered it even earlier and thought it was derogatory insult instead of a descriptive acronym: Create, Read, Update, and Delete. These four operations neatly encapsulates all fundamental functionality of a productive application (and even some entertainment ones, too.) This set of data manipulation activities drove HTML design from the very start: HTML <form> exist to enter and update data, and HTML <table> exist to present data.

Corresponding to those fundamental HTML 1.0 concepts were server-side mechanism that started with the Common Gateway Interface. Server-side infrastructure has evolved since their beginning, just as browsers have. By the time I started my web development education I had the luxury of platforms like Ruby on Rails. Which offered a “scaffolding” mechanism to automatically generate CRUD infrastructure so I didn’t have to write my own from scratch. Other development platforms offer similar counterparts, but that’s not the only way to go. I have since come across many more options for building a CRUD web app.

The simplest and easiest way into this world is Google Forms (and its competitors). I frequently encounter it for surveys, registration, etc. Which meant I always see the “Create” side of the app, though some forms allow me to return and “Update” my data. Whoever created the Google Form can then “Read” submitted responses and “Delete” if needed. Google Forms make creating a CRUD web app as easy as creating a Google Doc, and with better control over data than sharing a spreadsheet and telling people to add their data to a row.

I want to learn something quick and easy to use so I can use a database when it’s the right tool for the job. Minimize the hurdle of getting over the “ugh, I don’t want to spin up a database” hump. Google Form is a very simple way to go, but it goes too far: the data is output to a spreadsheet (Google Sheets) instead of a database perpetuating the “Excel is not a database” legacy. So I went looking for something between super simple Google Forms and full web development platforms like Ruby on Rails.

What I found are products that advertise themselves as “no-code” or “low-code” web app tools. The first one I came across was Amazon’s Honeycode under their AWS umbrella, but that has since been shut down. (I was going to link to the shutdown announcement, but it was posted to the Honeycode site, which is now dead.) Keenly reminded of the perils of putting my data into an online service, I focused on solutions I could run at home and found Budibase as a promising candidate.

Roger Cheng

Browser Based Database Front End

I have added WebUSB to my ever-growing to-learn list, hoping it would enable some really cool project ideas. As indicated by its name, I would need to know both web development and USB development before I can make them work together towards my goals. While I work my way through my self-directed USB study syllabus, I will also spend some time reviewing web development.

I’m not a complete beginner with web development, but that world is broad and it evolves quickly. Every time I applied browser-based technology to a project idea, I find some of my knowledge out of date. Plus I have to learn something new in an area I haven’t dealt with before. This is great! Approaching from a different direction every time helps me get a more well-rounded picture of the whole thing. This time I want revisit building web apps that present an user interface for a database.

My motivation is twofold. One is directly related to my desire to use WebUSB. Since any WebUSB project will be dealing with data flowing in and out of an external source (the USB peripheral) I expect many JavaScript constructs to resemble those used to communicate data to and from an external database. The second is that I’ve wanted to get better at applying database technology. So I can use it when it is the right tool for the job instead of, say, abusing Microsoft Excel. “Excel is not a Database” is an ongoing joke in the computer world. People do it because starting an Excel spreadsheet is far easier than setting up a database. Excel is “good enough” at small scales, but spirals into chaos as data set grows. There’s no shortage of horror stories ranging from Formula 1 race car construction to losing important healthcare data.

I want to avoid such disasters myself, because I’m definitely guilty of abusing Excel for poorly-suited problems. Some novel hacks turn out to be a delightful success, others times I find I really needed an actual database. I want it in my toolbox. This is something of a “return to where it all began” because my first web technology lesson many years ago came from The Ruby on Rails Tutorial. The main tutorial project was a web application interfacing with a database server. It’s a pattern common enough that I’ve since learned of an acronym for it: CRUD apps. And there exist many ways for me to build one of my own.

Roger Cheng

Potential WebUSB Study Syllabus

I followed through the steps of an Adafruit WebUSB example and established connection between Chrome browser on my Android phone to a microcontroller plugged into the USB port on the phone. I think WebUSB would enable many of my project ideas. But before I can turn any of my vague ideas into reality, I have a lot of homework to do.

The Adafruit example was thin on background information. I think it was written for people who already know how to work with TinyUSB library and just wanted to see Adafruit’s adaptation into an Arduino library. This impression is backed up by the fact its GitHub repository README (https://github.com/adafruit/Adafruit_TinyUSB_Arduino) is written using vocabulary I don’t understand. Following the link to TinyUSB’s site gives me similar language, so I have to start climbing my learning curve from somewhere else.

Searching from the browser side, I found a Chrome developer documentation page on WebUSB. I was able to comprehend more of this page, but not all of it. Here I learned one constraint on WebUSB: web apps are only allowed to connect to USB devices the operating system doesn’t already have a driver for. This is a mitigation against malicious apps bypassing operating system protection for USB devices like security keys. It also avoids ambiguity/duplication with existing functionality. For example, there’s no real need for a web app to interface with a USB keyboard via WebUSB when the operating system can already deliver key press events. Though there’s an interesting wrinkle here around USB serial, a common way to connect to microcontroller projects. By this rule, a web app can’t connect to a USB serial device on my desktop via WebUSB because my operating system already knows how to work with a serial device. (So it’s over to Web Serial land.) But apparently Android lacks a built-in handler for serial, so maybe it’s available via WebUSB? At this point I don’t yet know if that’s an opportunity or just a source of confusion.

Fortunately for beginners like myself, author of this Chrome developer documentation page included a link to USB in a NutShell for those unfamiliar with fundamental USB concepts. Hey, that’s me! I will try to start with USB in a NutShell and work my way back to TinyUSB and its various incarnations like Adafruit’s Arduino library. But that’s just the “USB” part, I have a lot I’ll have to learn for the “Web” part as well.

Roger Cheng

Adafruit WebUSB Arduino Example

I knew WebUSB existed but not much more than the name implying some sort of USB capability for web applications. I was motivated to look into it a bit more after learning about an index of browser diagnostics tools. Initial inconclusive signs were not promising but I kept looking. After a bit I thought: I want to explore this capability for my electronics projects. Maybe somebody at Adafruit has already looked into this? Searching for WebUSB on Adafruit Learn gave me a hit: Using WebUSB with Arduino and TinyUSB. So yes, they have!

I was happy to see the hardware in this example was Adafruit’s Circuit Playground Express (3333), because I have one already on hand. When following an example for the first time, it’s always nice to have the exact same hardware that I know will work, rather than similar hardware that should work. Despite that advantage it was not smooth sailing. I got stuck when it came to changing my Arduino IDE’s Tools/USB Stack to “TinyUSB”: there wasn’t a “USB Stack” option under “Tools” menu! I ran around in circles for a while before I eventually figured out I was using the wrong Arduino board support package. This example required “Adafruit SAMD Boards” and not “Arduino SAMD Boards”. I was thrown off because “Arduino SAMD Boards” included support for a bunch of Arduino boards and Adafruit’s Circuit Playground Express. I was able to select the proper board without realizing I was in the wrong board support library. I don’t know why Arduino claims support for boards that aren’t theirs, when the manufacturer has provided their own board support. It’s confusing, this is the second time they bit me, and I’m not happy.

Anyway, once I installed Adafruit’s board support library and selected Circuit Playground Express under Adafruit’s umbrella, I had a “USB Stack” option under “Tools” and could proceed to follow along with the example with no further issues. My first run used Chrome on my desktop computer, and after that success I tried it with Chrome on my Android phone. It works there, too!

And I can verify chrome://device-log is no longer empty on the phone, it now shows the newly-connected USB hardware.

This is huge! WebUSB might enable many project ideas that involve using one of my retired Android phone as the display (or more) of an electronics project. Which ones? I won’t know for sure until I learn more about the constraints of Android Chrome WebUSB support. I would have to pick a relatively simple one as a starting point before jumping into the more complex ideas. There’s a lot of study ahead. This Adafruit example was unfortunately lacking on background and theory of WebUSB so I’m on my own. I think it was written for people who already have the appropriate background, and that’s not me. Well, not yet. I need a refresher course on web development, and I will need to learn technical details of USB as well.

Roger Cheng

Android Chrome Device Log Strangely Empty

Learning about Chrome’s index of special URLs was very interesting. Aside from satisfying curiosity, it also gave me the tools to investigate an idea: can I write a web app to use an Android phone as interface to an electronics project that communicates over USB?

I want to repurpose my old retired Android phones as project UI, and have been making small incremental steps. My AS7341 spectral color sensor project presented its data as a web page served by the ESP32 on board, and my Android compass app for magnetometer exploration was also a web app to visualize data from my phone’s onboard sensors. But I haven’t been able to combine a phone’s onboard capability with external offboard capability. The barrier is a security measure: only web apps served from public TLC-secured https address is allowed to access extended capabilities like magnetometer. Web apps served locally over unencrypted http, like those served by my ESP32, is not allowed to access such things.

At one point in the past, web apps served via secured https was allowed to retrieve data from non-secure http sources, but I found that has been locked down in modern browsers. Now they require https all the way. I found this restriction during research for an earlier iteration of my AMG8833 thermal camera idea: I thought I could pair AMG8833 data with a phone’s onboard camera, but the https/http barrier sunk that plan. I had to wait for my Adafruit Memento to revisit that idea.

WebUSB is another one of these https-only features. If I can communicate with external peripherals over WebUSB, I can serve a web app from a https source (like GitHub pages) and talk to my hardware over USB instead of forbidden insecure http. To test this hypothesis, I took a USB keyboard and plugged it into my desktop PC running Google Chrome. I brought up chrome://device-log to verify that a USB keyboard shows up as a newly attached HID peripheral.

I then plugged the same keyboard into my Google Pixel 7. The keyboard is recognized and functional: I brought up Google Chrome and could type chrome://device-log. But unlike Chrome on my desktop, Chrome on my phone does not show a newly attached USB keyboard as HID peripheral. It just shows a completely empty device log. I know that even if a device shows up here it is not a guarantee that it supports WebUSB. But it’s not very promising when the log shows nothing at all. Does this necessarily mean Android Chrome doesn’t even see the hardware? That would be discouraging.

I know USB doesn’t work the same way on an Android phone as it does on a PC. For one thing, Android control panel has this “USB Preferences” screen to control how my Android phone uses its USB port. This screen represents a mechanism unique to Android USB behavior. There may be others, and I’ll have to learn to work with them. I checked https://caniuse.com and it says WebUSB is supported on Chrome for Android. That encouraged me enough to keep searching for more information on how this might work and found a WebUSB example from Adafruit which managed to make my Android device log less empty.

Roger Cheng

Looking Under A Browser’s Hood

Every aspiring web developer quickly learns about the collection of developer tools built into a browser, where we can look at how the browser has interpreted the HTML/CSS/JavaScript for the current page. It’s a very powerful set of diagnostic tools, but they’re about the content. What about the browser itself? There’s an entirely different set of tools to look at how the browser itself functions.

I’ve seen bits and pieces of these behind-the-scenes browser configuration and diagnostics pages before. Like when I had to enable Android Chrome’s compass API, or switch Firefox PDF viewer to full page zoom default. While I didn’t know the full extent of what’s available, I had assumed there would be a central directory buried somewhere in their respective developer documentation. I was half right: there is indeed a directory, but it’s not buried in documentation, it’s accessible right in the browser itself.

For Google Chrome, the master index is chrome://chrome-urls/. I learned of this from an old Tweet by Massimo Banzi (a.k.a. Mr. Arduino) who pointed to chrome://device-log/ and from there to the master list of all similar URLs. chrome://device-log/ shows all USB-related activity visible to Chrome and helps diagnose problems with things like WebUSB and WebSerial. It’s a sensible introduction for someone deep into the Arduino world.

But chrome://chrome-urls/ opens up a much broader view into Chrome browser internals. I was fascinated by chrome://omnibox/ which shows what happens when we type into what was originally Chrome’s address bar but had grown into so much more. How does Chrome choose to interpret what I typed as an address, or as a search term, or if I’m trying to go back in my browser history? chrome://omnibox/ shows all. And that’s just one of many special URLs listed by chrome://chrome-urls/.

For Microsoft Edge, which is now a Chromium-based browser, it has its own counterpart with search-and-replaced name edge://edge-urls. Mozilla Firefox went to a slightly different route, with their index available at about:about. I didn’t find an equivalent index for Apple Safari. Maybe there isn’t one built-in like the rest? I don’t use Safari as much anyway.

Roger Cheng

Introduction to Electronics With Tom Thoen

My personal path of electronics education was not a planned curriculum. It was a meandering haphazard collection of lessons I learned by browsing, observing, and screwing around. Absorbing knowledge as I go whenever and wherever I can. So when someone asks how they can get started, I didn’t know where to point them. “Visit Radio Shack some decades ago, before they sucked” is not a useful pointer. Eventually I learned of Adafruit and all the resources they put online for beginners, and have been pointing people in that direction. But I got a Radio Shack flashback when I read A 65-in-1 the 2024 Way on Hackaday.

I remember that type of educational electronic kits on Radio Shack shelves and remember wanting one of my own, but I never did. I’ve talked to many people who fondly remember their kit and credit them with changing the trajectory of their life. Pretty amazing and occasionally I wondered if getting one would have significantly changed my own life. I’ll never know the answer, but I’m glad somebody is working to recreate that kind of magic for a new generation of potential tinkerers.

Following the link on that page, I learned the project creator Tom Thoen has been doing more than just nostalgically recalling his old kit. He has instructed introductory electronics courses at local colleges and his class materials are available online at https://www.profthoen.com. Reading through the lecture slides by themselves isn’t as good as getting it alongside their accompanying lecture, but still informative. I think this is a great addition to the collection of pointers I can give to aspiring beginners.

[Header image: Main logic board from my Form 1+ teardown]

Roger Cheng

Scratching The Surface Of Autofocus

A camera’s auto-focus algorithm is a bit of black box magic where I have a passing interest. I was disappointed to learn while Adafruit’s Memento camera module exposes programmable control over its focus mechanism, the auto-focus algorithm remains opaque hiding inside its own black box. A brief survey of custom Canon camera firmware found that they’re not likely to have any details, either, as they call into existing Canon code to handle that kind of thing. A bit of web search found that I could dig up research papers on the topic, but I don’t foresee how this interest could grow beyond idle curiosity.

With commercial offerings at a dead end, I started looking at the venerable OpenCV library and how it might enable research and experimentation in auto-focus algorithms. It turns out Adafruit Memento is not the only camera to expose control over focus mechanism or triggering an auto-focus black box. Some cameras expose similar things in their OpenCV capture driver in the form of capture capabilities. I suppose it would be possible to connect an Adafruit Memento in the same way and use it with OpenCV. I will file that away as a potential future project.

Beyond OpenCV drivers, I did find some research papers about auto-focus. This is great, because research papers usually include an overview section that describes prior work in the field. This helps anyone who wants to get up to speed. If they can find a few papers in the field, they can look over the citations list. Any common citations are likely to point to foundational work and a good place to start. I skimmed through my first two hits: Robust Automatic Focus Algorithm for Low Contrast Images Using a New Contrast Measure (2011) and a more recent paper Learning to Autofocus (2020) applying some machine learning to the problem. My comprehension was pretty low, I can tell I have a lot to learn before I can really follow these papers.

So there’s a lot of work… what about the reward side of the equation? Right now my Adafruit Memento is the only camera where I could even try to do anything hands-on. Running autofocus in my own software is going to be far slower than the existing hardware component. And what might I be able to do with such a thing that’s interesting and different from existing autofocus? Right now I don’t have any ideas, which means not much motivation, resulting in a poor work-to-reward ratio. Until a suitable idea arises to motivate me to dive in to the field, autofocus will remain a passing curiosity sitting at a low priority in my project to-do list.

Roger Cheng

Window Shopping Plex Alternative Jellyfin

I’ve been using Plex for a few years to run my home local network media server, with vast majority of usage centered around my music CD collection in the form of MP3 files. It’s a fairly simplistic usage pattern so I haven’t encountered any problems. Most of the problems I had with running Plex server was my own fault, because I also used Plex as a test case to explore various home server technologies. That’s not Plex’s fault unless you want to say “Plex makes so many different deployment methods available” is a fault. At the moment I am not motivated to move off Plex. But if I ever do, I recently learned of Jellyfin as an alternative.

I used Plex at its free tier for a while before choosing to pay the one-time lifetime subscription fee to unlock a few features that looked interesting. I ended not using most of those features, but that was fine. A company needs revenue to operate and I paid the company for a product I found useful so the current situation is fine. But like most businesses, Plex is constantly trying to expand its revenue stream. So far that’s been in the form of paid features like video rentals but it does make me a bit suspicious. Tech industry history has many tales of companies alienating their existing customers in pursuit of money. I would be disappointed (but not surprised) if one day Plex decide my one-time lifetime subscription should become more than one-time or less than lifetime.

I have my media on my TrueNAS server, and I am running Plex Media Server software on a virtual machine of my Proxmox server. All capability is on my home local network, but I have to periodically sign in to my Plex account to verify my subscription status. If something happens to be offline at this critical time I would be locked out of my own Plex server. This has actually happened a few times, but not (yet) often enough to motivate me to ditch Plex.

Jellyfin improves on both of those problems. It is a free software project so as long as it stays on that ethos, I shouldn’t fear my account status changing and getting charged more money. In fact, I should be able to run everything on my home network server and never have to authenticate against an account server on the internet. Jellyfin is a less mature project with fewer features than Plex, but it has the “keeping MP3 collection organized” feature already and that’s most of what I need. If Plex (either the business or the software) gives me enough grief, I will likely migrate to Jellyfin for my home local network media server needs.

Jellyfin logo is from their UX repository https://github.com/jellyfin/jellyfin-ux

Roger Cheng

Internet Search: Nonsensical Plagiarists

Out of curiosity, I took a look at some tools provided by search engines to help sites drive traffic. I’m very glad site traffic isn’t important to https://newscrewdriver.com because search engines ask for some weird things. On the other hand I appreciate that making a good internet search engine is a constant battle against unscrupulous people trying to game the system. There are some legitimate things sites can do under the umbrella of “search engine optimization” but there’s a lot of deceptive SEO techniques out there as well.

This site is my personal project notebook. I write things down as I go, and it’s all searchable whenever I want to look back on something. It’s publicly visible in case someone is willing to sift through a lot of my ramblings to find a nugget of gold useful to them. Unfortunately it also means it’s publicly available for others to copy for less noble purposes. There’s a lot of those! When I want to search my own project notebook, I have to restrict it to my own site or I’ll get a lot of mangled junk copies.

Here’s one example: several years ago I was playing around with PIC microcontrollers and using Microchip’s MPLAB development tools. When I moved from browser-based MPLAB Xpress to desktop MPLAB X IDE, I wrote down a few thoughts about how the move affected my ability to use git source control: MPLAB Xpress vs. MPLAB X: Git Source Control. Recently I started using another tool that wouldn’t necessarily work with git (more on that in a future post) and searched for my old post.

My default search engine is DuckDuckGo, which subcontracts out to Microsoft’s Bing. I repeated my search on both sites to confirm this observation: If I search on the title of my blog post, both of those search sites’ first page of results include at least two hits from sites that copied my content. And there are more copycats in second page, third page, etc. You know what’s not on the first few pages of search results? My original page.

I blurred the URLs out of my screenshot above because I am not going to help drive traffic to those sites and I’m certainly not going to click on those links. But preview snippet shown by DuckDuckGo/Bing is enough to show me they’ve changed a few words in order to avoid being an exact copy. There’s a Roman numeral obsession where “MPLAB X” became “MPLAB ten” (uhhhh I guess it might be?) and “C source code” became “century source code” (no, absolutely not.) There was a misguided attempt to spell check when “git source control” became “get source control”. But the most puzzling changes involved things like taking my “and” and replaced it with “or” (or vice versa) which drastically changes the meaning of my original sentence. Their search/replace algorithm is not just ignorant of technical terminology, it’s just plain dumb at English syntax. I don’t know how those sites became ranked higher than my original text, and I’m not sure what purpose they aim to serve. I’m just sad at the possibility someone would be led astray in their own project because they found a mangled version of my project notebook.

Roger Cheng

Internet Search: Google Always Knows Best

One advantage of having my own domain like https://newscrewdriver.com is that I can subscribe to search engine services that provide feedback on how they process a site. Microsoft has Bing Webmaster Tools, Google has Google Search Console, etc. These are products focused on helping websites improve their chances of showing up in internet search results. An important objective for many sites, but not this one because I’m not trying to maximize revenue or anything here. I set up my dashboards because I was curious what information was available and what search giants expect site owners to do about it. Not that I can, for the most part, because majority of site implementation details here are handled by WordPress and out of my direct control.

Every once in a while I would receive a notification that some problem prevented a part of my site from search indexing. Most of these were caused by a change in WordPress and are quickly fixed with no action on my part. Sometimes the notification reflect a big attitude change in how Google thinks about the web and want sites to follow along. For example, a few years back Google decided a site’s treatment of mobile devices is more important than desktop, so sites are encouraged to have a good mobile experience or else their Google search ranking would take a hit. This was a change I agree with: making sites usable on small screens with slow processors and low bandwidth makes the web more accessible to all, and wielding Google search ranking as a stick to encourage adoption is one way to use Google power for good.

But then there are times that make me… less fond… of Google. I just received an error notification: “Duplicate, Google chose different canonical than user” with a link to this explanation. As I understand it, the error says that Google looked at the URL listed in my WordPress-generated site map and decided a different URL was better. I thought that was odd, but whatever. I went looking for a way to inform Google their decision was wrong, and I can’t. Google has decided they know my site better than I do and the “fix” is to change my site to use their chosen URL. Conform to Google. There is no alternative.

In this specific case, the URL was to one of my posts and Google’s chosen URL was to the comments section on that same page. Google is wrong, their link is not the superior link, but they will not hear arguments on their decision. Stalemate. I guess Google will keep using their wrong URL. Looking on the bright side, at least that URL actually points to my content, unlike some other search engines.

Roger Cheng

First Few Weeks With Dell Latitude 9410

Shortly after I bought an off-lease Dell Latitude 9410, I opened it up to verify everything met expectations. After finding no deal-breakers, I buttoned it back up and I’ve been using it for a few weeks. I’m happy with my purchase, here are some notes:

Intel Core i7

The Core i7 processor in my 9410 gives me snappy and responsive performance when I want it, then throttle itself down for low power consumption when I don’t. This means I get useful battery life on par with power-frugal Intel Atom machines, yet never feel sluggish during demanding times like I would with an Atom-based machine. My typical usage pattern results in 4-6 hours of runtime even with just 80% charge (more on that below) which ranks favorably among my past Intel-powered laptops. However, it falls short of my Apple MacBook Air with M1 Apple silicon. Both machines deliver snappy performance, but the M1 never gets as hot and runs far longer on battery.

The power and heat situation is a tradeoff against benefits of having an Intel CPU. All my developer tools are available here. I can run Docker containers without worrying about whether I need to find an ARM64 build. And pretty much every random USB peripheral will have Windows drivers. The biggest stumble so far is dual-booting Windows/Ubuntu: Dell configured this machine with Intel RST, and Ubuntu chose not to play well with RST. There are several possible solutions to this problem. I just haven’t been motivated enough to implement any of them yet.

Tablet/Laptop Convertible

Another thing my MacBook Air can’t do is fold its screen around and turn into a tablet. This was a deliberate design decision by Apple, who chose to keep MacBooks differentiated from iPads. After living with a Windows convertible for a few weeks I’ve decided I’m a fan. I’ve used Windows convertibles before but they’ve all been budget machines with limited hardware that hampered my user experience. This was my first full-power convertible and it means I can finally enjoy the benefits of a transformer and not trip over tradeoffs at every turn.

It’s nice to be able to switch back and forth. This is most useful when I’m reading documentation for software tools. I can switch to laptop mode to type a few commands for a quick hands-on exercise, then return to tablet mode and continue reading. With this positive experience I am much more receptive to such machines in the future, but I’m still not willing to pay the large price premium usually associated with such capability. Buying off-lease secondhand machines is likely to remain my pattern.

Power And Charging

Like my M1 MacBook Air, the 9410 charges via a USB Type-C connector instead of a proprietary power plug. However, this doesn’t necessarily mean I can use any USB-C power source. When up and running it demands 60 Watts. (USB power meter says 20V @ 3A.) This is even more demanding than the M1 MacBook Air, which is happy with 45W. If I plug in a less powerful USB-C supply, the MacBook will alert me that battery is still draining though at a slower rate. In contrast, this Dell would refuse to accept any power at all.

Fortunately, there are two charging systems on every laptop: there’s an operating system driver when the laptop is on, either running or in low-power sleep mode. Then there’s a completely separate firmware-based mechanism when the laptop is off, either in hibernation or completely shut down. Dell’s firmware-based charging system is willing to accepting power from sources that can’t deliver 60W. So if the bundled 60W power adapter fails or if I lose it, I still have alternatives.

And finally: like my Dell Inspiron 7577, there is a BIOS setting for me to restrict battery charging percentage. I set mine to stop charging at 80%. That still deliver enough battery runtime for most of my usage sessions, and avoiding charging to 100% should improve battery longevity. I can always change that setting back to 100% if I ever need extra runtime.

Whether iPad?

I justified this purchase as an alternative to upgrading my soon-to-be-obsolete 6th gen iPad and for my usage it was a success. At the moment I can imagine only two reasons why I might still want an iPad. The first is weight. Three pounds is light for a laptop but that’s triple the weight of an iPad. In practice, this hasn’t been a huge problem as I don’t like holding things up by hand for long periods of time, whether one pound or three. The second differentiator is the Apple app store, but at the moment I don’t need any iPad app exclusives. Maybe something else would arise as motivation for spending several hundred dollars? Until I encounter such motivation, I expect to be well-served by this Dell Latitude 9410.

Roger Cheng

Dell Latitude 9410 Internals: M.2 2280 Confirmed

Buying an used off-lease computer means giving up shiny-new cosmetic perfection in exchange for a hefty discount, a tradeoff I was willing to make. I don’t think I’ll be bothered by the “Grade B” blemishes on my unit and besides, it’s what’s inside that really counts. Purchasing from Dell Financial Services via their retail site https://dellrefurbished.com includes a 100-day warranty to back up their claim that all machines are fully functional regardless of cosmetic state.

My order confirmation email had a pleasant unexpected bonus: it included the Dell service tag for the specific machine I had just bought. I could then put that service tag into Dell’s support site to learn information about that specific unit before it had even arrived. There were still a few weeks left on its original 3 year warranty, though it would expire before the 100 day refurbished warranty would. Another positive attribute was the fact my unit was not equipped with a cellular data (WWAN) module. I don’t expect to use WWAN, and without the module, the machine should have room for a M.2 SSD in the more common, longer, and cheaper 2280 form factor. The factory configuration list included a model number for the factory SSD, which is associated with a M.2 2280 drive.

The machine arrived a few days later and, once I confirmed the machine functioned as expected, I brought up Dell’s service manual and opened up my unit to familiarize myself with its internals and to confirm information listed on Dell support. Compared with my previous Dell laptops, there was much more extensive use of thin adhesive-backed sheets of various materials. Are they for RF shielding? For airflow management? Other purposes? It’s hard for me to tell but their presence is not surprising in a device designed to be thin and light. I just have to keep in mind I can only remove them a few times before their adhesive gives out.

Modern component miniaturization allowed smaller circuit boards, freeing up more internal volume for the battery. Which needs to be disconnected and the system depowered before I disconnect anything else. This battery plug was very securely fastened and difficult to remove. Far more difficult than any previous Dell laptop battery connectors I’ve encountered. Hard enough that I triple checked I didn’t overlook some other mechanism I was supposed to release before unplugging the connector. But there were no other mechanisms, it was just a really tight fit.

After the system was depowered, I quickly made my way to the SSD to confirm it was indeed a M.2 2280 unit. This will make future upgrades easier and cheaper than the less common M.2 2230 type used in WWAN-equipped units. Speaking of which, I don’t think I can (easily) retrofit mine with one. The module connector is there on the logic board, but I don’t see any loose wires that would be appropriate for plugging into a cellular modem. So my machine probably lack cellular antennae as well. Though if I ever come into possession of a M.2 2230 SSD in the future, I might be tempted to give it a shot anyway for curiosity’s sake. Compatible WWAN modules seem to cost about $30-$50 from various Amazon vendors and maybe I can rig up a less elegant antenna. As long as I keep my expectations modest for such a project, it might still be an interesting data point. In the meantime I’m content to use the machine as-is.

Roger Cheng

Dell Latitude 9410 Cosmetic Grade B

I’ve decided to buy an off-lease Dell Latitude 9410 from Dell Financial Services, via their retail web site https://dellrefurbished.com. All of the machines have been evaluated to be in good functional order, but some of them have cosmetic blemishes separated into cosmetic grades. Cosmetic grade A are for machines in good shape, and grade B indicate machines that are in… less good shape at a lower price. Since I’m a cheapskate, I ordered a grade B unit and in my specific case, it wasn’t bad at all!

When the machine arrived, my first surprise was the label at the bottom: “Refurbished to Dell specifications by FedEx Supply Chain” I expected the evaluation and refurbishment process to be done by a Dell subcontractor, I just didn’t expect to see the FedEx name. Prompted by this surprise, I did a bit of research to find FedEx TechConnect, with uncertain relationship to “FedEx Supply Chain” formerly GENCO. To me it seems like an odd side gig for FedEx to take on, but I’m not a MBA at FedEx business development.

I found some damage on the keyboard. It means the backlight would shine through these damaged corners, not something I’d notice while I’m typing and looking at the screen. This damage is not a surprise in hindsight: when the screen is flipped around to turn this into a tablet, its keyboard becomes the exposed bottom of the device. The previous user of this laptop must have set the tablet down on something that caused this key cap damage.

Another problem with this device was the rubber strip at the bottom: it’s gone. There’s supposed to be a layer of light gray soft material overmolded onto this hard black plastic core strip. With the soft layer gone, all I have is this ugly looking strip. Fortunately I don’t have to look at it when I’m using the laptop. As a substitute, the refurbish process added cheap square stick-on rubber feet.

The stick-on squares are much thicker than the missing rubber strips. In laptop mode, this meant a larger gap at the bottom for better air cooling. But the thickness gets in the way when I fold the screen around for tablet mode: I couldn’t fold the screen completely flat when these thick rubber pads are in place. I will look for slightly thinner stick-on rubber feet to replace these thick squares, with the goal of restoring full tablet mode form factor while preserving laptop mode air cooling. I consider this a minor detail that is within my ability to fix, and not a big deal.

I found no noticeable damage on the screen, the metal body, or lid. Those were bigger concerns buying “Grade B” and my unit is practically pristine on those fronts. I don’t think the slightly scratched keyboard would bother me very much, and I can replace the stick-on rubber feet. I’m perfectly happy accepting those blemishes in exchange for >80% discount off original MSRP, especially when its internals look perfectly fine.