Notes on “Using MongoDB with Node.js” from MongoDB University

Most instructional material (and experimentation) for MongoDB uses the MongoDB Shell (mongosh), which is “a fully functional JavaScript and Node.js 16.x REPL environment for interacting with MongoDB deploymentsaccording to mongosh documentation. Making mongosh the primary command line interface useful for exploration, experimentation, and education like on Codecademy or MongoDB University.

Given the JavaScript focus of MongoDB, I was not surprised there is a set of first-party driver libraries to translate to/from various programming languages. But I was surprised to find that Node.js (JavaScript) was among the list of drivers. If this was all JavaScript anyway, why do we need a driver? The answer is that we don’t use JavaScript to talk to the underlying database. That is the job of BSON, the binary data representation used by MongoDB for internal storage. Compact and machine-friendly, it is also how data is transmitted over the network. Which is why we need a Node.js library to convert from JSON to BSON for data transmission. I started the MongoDB University course “Using MongoDB with Node.js” to learn more about using this library.

It was a short course, as befitting the minimal translation required of this JavaScript-focused database. The first course covered how to connect to a MongoDB instance from our Node.js environment. I decided to do my exercises with a Node.js Docker container.

docker run -it --name node-mongo-lab -v C:\Users\roger\coding\MongoDB\node-mongo-lab:/node-mongo-lab node:lts /bin/sh

The exercise is “Hello World” level, connecting to a MongoDB instance and listing all available databases. Success means we’ve verified all libraries & their dependencies are install correctly, that our MongoDB authentication is set up correctly, and that our networking path is clear. I thought that was a great starting point for more exercises, and was disappointed that we actually didn’t use our own Node.js environment any further in this course. The rest of the course used the Instruqt in-browser environment.

We had a lightning-fast review of MongoDB CRUD Operations and how we would do them with the Node.js driver library. All the commands and parameters are basically identical to what we’ve been doing in mongosh. The difference is that we need an instance of the client library as the starting point, from which we could obtain object representing a database and a collection with it. (client.db([database name]).collection([collection name]) Once we have that reference, everything else looks exactly as they did in mongosh. Except now they are code to be executed by Node.js runtime instead of typed. One effect of running code instead of typing commands is that it’s much easier to ensure transaction sessions complete within 60 seconds.

For me, a great side effect of this course is seeing JavaScript async/await in action doing more sophisticated things than simple straightforward things. The best example came from this code snippet demonstrating MongoDB Aggregation:

    let result = await accountsCollection.aggregate(pipeline)
    for await (const doc of result) {
      console.log(doc)
    }

The first line is straightforward: we run our aggregation pipeline and await its result. That result is an instance of MongoDB cursor which is not the entire collection of results but merely access to a portion of that collection. Cursors allow us to start processing data without having to load everything. This saves memory, bandwidth, and processing overhead. And in order to access bits of that collection, we have this “for await” loop I’ve never seen before. Good to know!

Many Paths to MongoDB Shell (mongosh)

To promote their product, MongoDB has setup their own online learning resource MongoDB University. I was curious to learn more about a database that offers something different from a standard SQL relational database, so I started with their “Introduction to MongoDB” course. Since it was at least partially a marketing tool, I was not surprised the course wanted to take us through a grand tour through all MongoDB products from their cloud-hosted MongoDB Atlas data platform to all the tools we can download and install. I was ready to just ignore and skip over those sales pitches, but then the course would quiz me to make sure I’ve actually installed them on my computer. This annoyed me, especially for the MongoDB Shell (mongosh). It’s how Codecademy introduces us to MongoDB, and it’s what we use in MongoDB University’s Instruqt hands-on labs. There are so many ways to get mongosh I refuse to download and run a full blown application installer package just for a command line tool.

At a minimum, this duplicates work. MongoDB Compass is another item the course wants us to download and install. Compass is a separate application that offers GUI-based methods for interacting with a MongoDB database and/or Atlas cluster. GUI are nice but rarely cover 100% of all scenarios, so developers like the option of dropping to a command line. Which is why clicking at the bottom of MongoDB Compass would bring up an integrated mongosh.

I’ve been using Docker as a tool to avoid installing software directly on my computer. I couldn’t escape installing MongoDB Compass because of its graphics interface, but as a command line tool mongosh is easy to run through Docker. The easiest way is to use the official MongoDB Docker image, which includes mongosh alongside the core database engine.

docker run --rm -it mongo:latest mongosh [connection string]

Doing this means we’re pulling down the entire MongoDB database just for the little mongosh tool. That’s like ordering an entire seven-course meal to eat just the little cherry on top of ice cream dessert. Even in this age of broadband internet, that seems rather excessive. I thought it’d be neat to try setting up a container just for mongosh, see if that’s any smaller.

I found instructions for installing mongosh on an Ubuntu instance. Ubuntu Focal is one of the supported versions so I’ll start there.

> docker run --name mymongosh -it ubuntu:focal

I’ll need some tools not found in this basic Ubuntu container, so I need to populate the package index followed by their installation.

> apt update

> apt install wget gnupg

After that I could follow MongoDB instructions. Slightly modified by removing “sudo” as it was unnecessary: we are running as root in this little world.

> wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | apt-key add -

> echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list

> apt update

> apt install mongodb-mongosh

And voila! I have a docker container “mymongosh” based on the Ubuntu Focal container. Let me see how big it is:

> docker ps --size
CONTAINER ID   IMAGE          COMMAND   CREATED          STATUS          PORTS     NAMES       SIZE
757519a645f8   ubuntu:focal   "bash"    19 minutes ago   Up 19 minutes             mymongosh   278MB (virtual 351MB)

Wow, getting this far meant pulling 278MB on top of 73MB of Ubuntu Focal. This was far larger than I had hoped. While this size still compared favorably with MongoDB image size of almost 700MB, I don’t think my little experiment was worth the effort.

PS C:\Users\roger\coding\PostgreSQL> docker image ls
REPOSITORY                                                        TAG              IMAGE ID       CREATED         SIZE
mongo                                                             latest           2dd27bb6d3e6   7 days ago      695MB

If I want small size, I’d have to learn how to build a minimal Docker image based on Alpine, which isn’t what I want to focus on right now. My next objective is to learn how to use MongoDB from Node.js code instead of mongosh.

Notes on “Introduction to MongoDB” on MongoDB University

I started learning about MongoDB on Codecademy, but there were technical difficulties blocking my progress. So I switched to MongoDB University, the free online learning platform hosted by MongoDB themselves. There was a bit of a learning curve on idiosyncrasies of MongoDB’s learning platform, but I got the hang of it soon enough and could focus on the information covered by “Introduction to MongoDB” course.

Each section of the course starts with a Wistia-hosted video. I prefer to learn by reading at my own pace over watching a video, so this wasn’t my favorite. The video sometimes has captions allowing us to read along with the spoken narrative, but the presence of captions was inconsistent. Sometimes the video is followed by a page of text covering the same information and I thought I could read that text and skip the video, only to find that sometimes the text is missing information covered by the video and vice versa. I end up having to do both video and repetitive text, which feels like a tremendously inefficient way to do things.

Another inefficiency are the hands-on labs powered by Instruqt. It takes several clicks and a few tens of seconds for the lab environment to spin up, which isn’t bad by itself except most of our exercises involve typing out a single mongosh (MongoDB Shell) command and exiting to proceed to the next lab. This implementation means we spend a lot of time twiddling our thumbs waiting for spin-up and tear-down overhead, I would have preferred that we do more in each Instruqt session, so we don’t spend so much time waiting.

As for the course material, it’s not technically just about the MongoDB database itself, the course is also a sales pitch for associated MongoDB products. Mainly MongoDB Atlas, the cloud-hosted (your choice of AWS, Azure or GCP) MongoDB service that cost money for us and generates revenue for the company. I don’t mind a company making sure we know how to give them money, but I don’t care for the fact that MongoDB Atlas features are intermingled with MongoDB core functionality in this course. When we run our own MongoDB instance (which I plan to do for my own projects) we won’t have Atlas functionality, but this course doesn’t always make clear what is core MongoDB versus functionality added by Atlas. I expect the company would say “Don’t need to bother, just use Atlas for your projects, we have a free tier!” and I appreciate that the offer exists. But I’m rather skeptical that their “Free Forever” tier will actually stay free given the discouraging historical record of free tiers.

Instructions in MongoDB mostly concentrate on what we can type into mongosh, which exposes a JavaScript style interface for interacting with a database. That is very different from storing SQL commands in strings as we did in node-sqlite. Compared to SQL syntax, I found it easier to remember and parse mongosh syntax because of its similarity to JavaScript. That said, I still come across things that feel inconsistent to me that catch me off guard. Some commands want us to specify an action first then where to store the results. {$count: "total_items"} Other commands want us to specify the location of results followed by the action to generate it. {"total_items": {$count {}}} Maybe there’s a system and I just haven’t learned them yet, certainly the course never tried to explain them.

At least I could still follow the logic for most of those inconsistencies. The one that completely confused me was an example using geographical location data in the form of { type: 'Point', coordinates: [ 40, -74 ] } I think this is their GeoJSON format but the course didn’t go into detail. The example then sorted by ‘latitude‘, and I don’t see how I could have looked at ‘type‘ and ‘coordinates‘ and decide ‘latitude‘ is available for sorting. To my eye ‘latitude‘ was not defined at all! If I figure this out later, I’ll add a URL here to the appropriate reference.

Here is my super-abbreviated variation of course syllabus:

  • Querying for data with db.[collection].findOne() and find(). Followed by insert with insertOne() and insertMany(). We start with a few basic query operators using comparison operators $gt, $gte, $lt, and $lte. Then logical operators like $and and $or. We can use $elemMatch if we want to peer into values in the form of an array.
  • Modifying data with replaceOne for direct replacement. updateOne() and updateMany() take operators like $set and $push. findAndModify() combines a common pattern of finding a single document, modifying it, and returning the results. Finally deleteOne() to remove documents.
  • Tailor query results with cursor.sort() and limit(). Projection parameter to find() lets us skip information we’re not interested in. And collection.countDocuments() is useful for data exploration.
  • Aggregation is roughly analogous to SQL table joins. We are introduced to operators $match, $group, $sort, $limit, $project, $count, $set, and $out.
  • Index helps us optimize lookup for specific query patterns. Recommend we list the fields in usage order of Equality, Sort, then Range. (But couse didn’t explain why.) Like SQL, there’s always some tradeoff involved for having a database index. The explain() command helps us see if an index is actually as helpful as expected.
  • There’s a whole arena of information on MongoDB anti-patterns. Like how we should be aware of BSON size limit of 16MB which means avoiding things like creating arrays that would grow unbounded.

This course gave me a solid start to using MongoDB in my projects. As I do not intend to rely on MongoDB Atlas staying free, I’m more likely to run the MongoDB Docker container on a test machine at home. Interactions with such a database would likely happen via mongosh and there are many ways to get it.

Notes on MongoDB University Learning Platform

I’ve been learning a lot of interesting things from Codecademy’s course catalog, including the fundamental concepts of the “NoSQL” database software design of MongoDB. However, when Codecademy’s hands-on learning mechanism got stuck, I couldn’t see how those fundamental concepts translated to MongoDB practice. But that’s OK, the Codecademy MongoDB course was built in conjunction with MongoDB themselves, so I can try their own MongoDB University platform for learning online.

The switch does introduce some friction, though. Starting with the online learning platform itself. Codecademy rarely uses video lectures. (Which I appreciate, and more comments about content will come later.) When they do, they embed a YouTube video and let Google figure out the rest. MongoDB University hosts their (more frequent) video lectures on Wistia. (“Where Video Meets marketing”.) Wistia’s video player component has a slightly different interface from YouTube. The most annoying aspect of the Wistia player is lack of memory across sessions. I prefer playing the videos slightly faster than standard speed, and its lack of memory means I have to click the settings button to speed up playback on Every. Single. Video. I also prefer to turn on English captions, which meant more clicking on every video. Not every MongoDB University video had captions available, but I don’t blame Wistia for that. I do wish they followed YouTube’s lead and offer at least the option of imperfect (but far better than nothing) auto-generated captions.

Since I loved Codecademy giving us hands-on interaction with the material, I was happy to see MongoDB University courses also has a hands-on interaction component powered by Instruqt. (“The #1 Hands-on Virtual IT Labs for Product-led Growth”) For these exercises, Instruqt provides a mongosh (MongoDB interactive shell) command line in our browser window. It feels like they’ve spun up at least two Docker instances for each exercise: one container running mongosh, connected to another container running MongoDB itself. This works well as we get a known starting state for each exercise and, after we’re done, undo anything we’ve changed so the next person gets the same starting state. The downside of a browser-based command line is that we don’t get the full set of command-line keyboard shortcuts. I had to use mouse right-click in order to copy and paste because my preferred keyboard shortcuts didn’t work.

Another similarity to Codecademy are quizzes sprinkled throughout the course to test our comprehension of course material. User interface for these quizzes leave something to be desired. The first problem is when I clicked “Show Results”, I saw items I didn’t select displayed as “Incorrect”. I was confused for a few minutes, reading the explanation trying to figure out where I went wrong. Eventually I figured it out: I wasn’t wrong. They’re just showing me all of the explanations. I did not select the incorrect answer and that was the correct thing to do. That was very confusing. The next problem is literally the “Next” button on these quizzes. After finishing one question, I clicked “Next” to proceed with the quiz, only to get disoriented when the course continued with new material. Eventually I figured out there is a “Next” button to continue with the quiz, and it is near a different “Next” button which will abandon the quiz and proceed with the course. I clicked the latter when I should have clicked the former. This was a pretty bad user interface design but once I figured out what was going on, I could deal with it and focus on the course material.

Notes on Codecademy “Learn MongoDB”

Codecademy’s course catalog on SQL databases is thinner than those on topics like web front-end development, and their in-browser learning infrastructure isn’t as polished for those courses either. This has caused me frustration, but I was still learning useful knowledge. Codecademy’s PostgreSQL skill path was packed with information I can use and links to where I could learn more. After that, though, I wasn’t interested in anything else under their SQL umbrella of courses, but that doesn’t mean I’m done with databases because SQL relational databases are no longer the only game in town: A few alternative “NoSQL” database designs have recently arisen, and I had been curious about their design tradeoffs. So, the next step is Codecademy’s Learn MongoDB course, which I learned was launched (or at least promoted) recently via a Codecademy mailing list. Unfortunately its technical immaturity caused problems, more on that later.

This course started well, with a review of databases for those who come into the course without a background in SQL relational databases. With that background established, it proceeded to describe how NoSQL databases (there are several subtypes) like MongoDB (representing the “document database” subtype) go about their business. I loved this section, because it answered my question about why these databases exist and when they might (or might not!) be the right tool for the job.

On the course syllabus it said “Built in partnership with MongoDB” which in practice meant many links to MongoDB’s established portfolio of guides and documentation. After Codecademy’s own explanation of SQL vs. NoSQL, we have a link to MongoDB’s own take. Related to that topic is a presentation that describes MongoDB structures in terms of close analogues in SQL, but also implored experienced database developers to free their mind and think beyond relational database conventions. It seems perfectly possible to set up a MongoDB database so it looks and act like a relational database, but not taking advantage of MongoDB strengths risks incurring all the disadvantages of NoSQL without any reward to balance them out.

The MongoDB advantage that really caught my attention is the ability to start working on a project before we know everything about data access patterns. In a SQL-backed project that is a recipe for disaster because incomplete information would lead to a suboptimal schema, and one that we’d be stuck with towards the end of the project. Over in MongoDB land, our data validation can be loose in early stages of the project and tightened as we go if desired. During the course of development, MongoDB can theoretically adapt to changes much more easily than SQL databases could handle schema updates. I wonder if this means it’s possible for an application to evolve their MongoDB usage and end up in a state where it’d make sense to migrate to a SQL relational database.

These features were all very promising, and I really looked forward to playing with MongoDB. Unfortunately, Codecademy’s hands-on lesson backend is broken today. On the first page of the first interactive lesson, I was told to show all databases in a MongoDB instance by typing in the “show dbs” command, which listed four databases as expected. After this list was shown, I was to click the “Check Work” button to verify my progress before proceeding. But when I clicked “Check Work”, nothing happened. I did not see a successful check, which would have allowed me to proceed. In the absence of success check, I expect to see a red X telling me my answer was wrong and need to try again. But I didn’t see that, either. Nor did I see any sort of an error message. No “Pass”, no “Fail”, and not even a “Oh no something is wrong”.

I’m stuck.

So Codecademy’s Learn MongoDB course was a bust, but as mentioned earlier, MongoDB has their own collection of learning resources. The reading material so far got me interested enough that I want to continue learning MongoDB. Instead of waiting for Codecademy to fix their backend, I will switch to MongoDB’s learning platform.

Notes on Codecademy “Design Databases with PostgreSQL”

After a frustrating experience with the node-sqlite course on Codecademy, I’ve concluded that their in-browser instruction environment is not well set up for teaching SQL. Or at least, this course is far weaker than others on Codecademy, which I had been generally satisfied up until this point. I considered switching to a different topic but decided there were still a few more things I wanted to learn. But from here on out, when I fail an exercise, I’m more likely to decide my answer was fine. (More willing to hit “View Solution” to see the answer and compare.) And more importantly, I’m going to review the course syllabus beforehand and skip those with frustrating “Code Challenge” sections.

Which led me to “Design Databases with PostgreSQL” which is technically a “Skill Path” offering rather than a course. Like my previous skill path in web development, I started this skill path and was immediately at 50% completion due to material that overlapped with courses I’ve already taken. One major difference is that all my Codecademy database courses used SQLite to illustrate general SQL concepts, but this skill path has some PostgreSQL-specific details amongst more general database concepts.

Initially, I was mildly disappointed that the material was shallower than I had expected based on course description. The sales pitch made it sound like we’re going to get into some real detailed nuts-and-bolts, but while the course did indeed to into further depth than other courses to date, there were still many topics that ended with “we’re not going into more depth, but at least now you are aware it exists.” An example concerns database keys. From my earlier courses I had known about primary, foreign, and composite keys. This course mentioned there are also super, candidate, and secondary keys then proceeded to say nothing more about them. As a starting pointer this is fine, I had just expected more.

Once I adjusted my expectations, this skill path was time well spent. We get more information on database schema and that proper design of a schema can make a difference in database performance. Both at development time (make queries easier and less prone to problems) and at runtime (easier updates and faster queries.) Part of this design process is database normalization, which was covered by starting with a poorly designed database. After covering how a poorly normalized database causes problems in use, we are walked through typical normalization techniques to solve those problems.

But those solutions usually have a tradeoff. This course has a recurring theme in the form of database tradeoffs. A competent database engineer has to be able to understand the problem domain and usage patterns to properly prioritize certain constraints versus others. A normalized database has a lot of space, performance, and consistency advantages. But it does tend to make updates and queries more complex by requiring database joins. Similarly, a database index can make queries faster, but maintenance means updates are slower. The index also takes up space on disk.

A light complaint I have about this course is that its illustrative examples were surprisingly poor. One example used an email address as a primary key for a list of people, but a person can have multiple email addresses making it a poor primary key. Another example separated ingredients from recipes, but the ingredient is associated with a fixed amount. It is unrealistic for a cookbook to use, say, the same amount of salt for every recipe.

One of the practice exercises was “Bytes of China”, setting up a database that would track information suitable for a restaurant menu. From the starting directions I had looked forward to an open-ended exercise, because it said to: “Create table with columns that make sense based on the description” This came to a screeching halt when we were given information to add to these tables in the form of SQL INSERT clauses that expect a specific database schema. I had to delete the database schema that made sense to me and rebuild a different schema to suit the exercise data. This was annoying. They could have told us to build to suit their sample data upfront instead of letting us waste time designing our own that wouldn’t work.

Gripes aside, I learned a lot of neat things that I expect to use in future projects that might need a database. I learned how it was possible to represent many-to-many relationships with a “join table” that has a composite primary key that consists of a combination of foreign keys. Beyond “PRIMARY KEY” I learned about constraints like UNIQUE, CHECK, and REFERENCES. In the earlier SQLite course I was dismayed to learn it doesn’t enforce the schema, calling it a flexible feature instead of a bug. PostgreSQL isn’t as free-wheeling but we still need to watch out for times when it becomes inadvertently unhelpful. If I make a mistake trying to put a floating-point number like 1.5 into an integer field, PostgreSQL would round it to 2 without error. Or if I make a mistake putting it into a text field, PostSQL would helpfully convert the number 1.5 to the string “1.5”.

Every bit of SQL instruction I’ve come across before only ever joined two tables, this course was the first to teach me how to join more than two. I can see how this would feel obvious to SQL veterans, but every beginner has to see it at least once:

SELECT table_one.column_one AS alias_one, table_two.column_two AS alias_two, table_three.column_three AS alias_three
FROM table_one
INNER JOIN table_two
ON table_one.primary_key = table_two.foreign_key
INNER JOIN table_three
ON table_two.primary_key = table_three.foreign_key;

I think all of the remaining nifty tricks are PostgreSQL-only, but I’m not sure. The course doesn’t make a lot of distinction between thing we can use in other databases versus PostgreSQL-only. So I don’t know if I can extract just the date from a timestamp (DATE_PART()) with other databases, or if I can make this query to examine constraints on record for a table:

SELECT
  constraint_name, table_name, column_name
FROM
  information_schema.key_column_usage
WHERE
  table_name = 'fill in table name';

Given the pg_ prefix, the following query is likely PostgreSQL specific way to list every index built for a table.

SELECT *
FROM pg_Indexes
WHERE tablename = 'fill in table name';

Also with the prefix is a query to show size of a table, which includes space consumed by storing data and all associated index.

SELECT pg_size_pretty (pg_total_relation_size('products'));

And finally, we get a few starting points for performance analysis. Like prepending EXPLAIN ANALYZE in front of a query to get information on how the database plans out its execution. Or SELECT NOW(); to print out a timestamp before and after an operation so we can see how long it took.

That’s a lot of information packed into a single Skill Path. I wished for more, but I can understand there’s a tradeoff against making the course too long. Maybe this is the perfect length after all, and just enough for me to learn more on my own later. I can spend years learning all the intricacies of relational databases, but right now I’m more curious to explore something a little different.

Notes on Codecademy “Learn Node-SQLite”

After my SQL fresher course, shortly after learning Node.js, I thought the natural progression was to put them together with Codecademy’s “Learn Node-SQLite” course. The name node-sqlite3 is not a mathematical subtraction but that of a specific JavaScript library bridging worlds of JavaScript and SQL. This course was a frustrating disappointment. (Details below) In hindsight, I think I would have been better off skipping this course and learn the library outside of Codecademy.

About the library: Our database instructions such as queries must be valid SQL commands stored as strings in JavaScript source code. We have the option of putting some parameters into those strings in JavaScript fashion, but the SQL commands are mostly string literals. Results of queries are returned to the caller using Node’s error-first asynchronous callback function convention, and query results are accessible as JavaScript objects. Most of library functionality are concentrated in just a few methods, with details available from API documentation.

This Codecademy course is fairly straightforward, covering the basics of usage so we can get started and explore further on our own. I was amused that some of the examples were simple to the point of duplicating SQL functionality. Specifically the example for db.each() shows how we can tally values from a query which meant we ended up writing a lot of code just to duplicate SQL’s SUM() function. But it’s just an example, so understandable.

The course is succinct to the point of occasionally missing critical information. Specifically, the section about db.run() say “Add a function callback with a single argument and leave it empty for now. Make sure that this function is not an arrow.” but didn’t say why our callback function must not use arrow syntax. This minor omission became a bigger problem when we roll into the after-class quiz, which asked why it must not use arrow syntax. Well, you didn’t tell me! A little independent research found the answer: arrow notation functions have a different behavior around the “this” object than other function notations. And for db.run(), our feedback is stored in properties like this.lastID which would not be accessible in an arrow syntax function. Despite such little problems, the instruction portion of the course were mostly fine. Which brings us to the bad news…

The Code Challenge section is a disaster.

It suffers from the same problem I had with Code Challenge section of the Learn Express course: lack of feedback on failures. Our code was executed using some behind-the-scenes mechanism, which meant we couldn’t see our console.log() output. And unlike the Learn Express course, I couldn’t workaround this limitation by throwing exceptions. No console logs, no exceptions, we don’t even get to see syntax errors! The only feedback we receive is always the same “You did it wrong” message no matter the actual cause.

Hall of Shame Runner-Up: No JavaScript feedback. When I make a JavaScript syntax error, the syntax error message was not shown. Instead, I was told “Did you execute the correct SQL query?” so I wasted time looking at the wrong thing.

Hall of Shame Bronze Medal: No SQL feedback. When I make a SQL command error, I want to see the error message given to our callback function. But console.log(error) output is not shown, so I was stabbing in the dark. For Code Challenge #13, my mistake was querying from “Bridges” table when the sample database table is actually singular “Bridge”. If I could log the error, I would have seen “No such table Bridges” which would have been much more helpful than the vague “Is your query correct?” feedback.

Hall of Shame Silver Medal: Incomplete Instructions. Challenge #14 asked us to build a query where “month is the current month”. I used “month=11” and got nothing. The database had months in words, so I actually needed to use “month=’November'”. I wasted time trying to diagnose this problem because I couldn’t run a “SELECT * FROM Table” to see what the data looked like.

Hall of Shame Gold Medal Grand Prize Winner: Challenge #12 asks us to write a function. My function was not accepted because I did not declare it using the same JavaScript function syntax used in the solution. Instructions said nothing about which function syntax to use. After I clicked “View Solution” and saw what the problem was (image above) I got so angry at the time it wasted, I had to step away for a few hours before I could resume. This was bullshit.


These Hall of Shame (dis)honorees almost turned me off of Codecademy entirely, but after a few days away to calm down, I returned to learn what Codecademy has to teach about PostgreSQL

Notes on Codecademy “Learn SQL”

I’m a little sad that hobbyist web app projects have lost the option of free hosting on Heroku, but that’s no reason to stop learning. Heroku is not irreplaceable, I’m sure I can figure out something if a project proceeds far enough to be worth the effort. So, back to learning: where should I go next? Looking at project ideas that involve Node.js and potentially Express, I decided the next area of focus is a backing datastore. It’s time for some database refresher work starting with Codecademy’s “Learn SQL“.

I’ve taken several database courses in the past, to varying levels of rigor and depth. I expected the introductory material of this course to be review so I’m better able to learn new concepts later in the course. As it turned out, this course was entirely review for me but to be fair, some concepts were fresher in my mind than others. I especially appreciated the cool animations illustrating various table joins.

This specific course could be more accurately titled “Learn SQLite” because that’s the database engine used in the course. Which is fine, it covers all the basics. The one thing I hadn’t known (or had forgotten) about SQLite is its… flexibility… in data types. It is standard operating procedure for SQL tables to be declared with a data schema. “Names are strings, IDs are numbers”, etc. While SQL was designed for the database engine to enforce this schema, SQLite does not. When the Codecademy course mentioned this, I said “What!?” but the assertion checks out, confirmed by SQLite’s own FAQ which declares type flexibility as a feature and not a bug. I come from a world of strictly typed programming languages like C, so flexible typing like JavaScript feels more like a problem waiting to happen than a feature. I feel the same with SQLite’s lack of schema enforcement.

Another reason to take a SQL refresher course now is to review all concepts from a new perspective. Now that I am thinking of using a database as backend storage for a web application. From this perspective, some of SQL features make less sense than in other contexts. For example, I’m not sure ORDER BY makes sense to do within the database engine, as a web app almost certainly needs to have sorting logic anyway. Think of the shopping sites that lets the user reorder by availability, by lowest price, etc. For small datasets I’d want to do that on the client end instead of round-tripping each new sort as a new query all the way to the database. But the story changes for large datasets. It’ll make sense to sort data on the database if we want things ordered and then LIMIT to the top X items. That reduces bandwidth consumption between server and client and would be a good tool to have.

In contrast, other features like CASE (to categorize values), AS (to rename columns), and ROUND (rounding numbers) are definitely tasks better performed on the client end. I can’t think of a scenario (yet) where it makes sense to do that work on the server-side database.

This course touches on the concepts of primary keys and foreign keys, but other than uniqueness we didn’t get any further details of relational database design. This course didn’t cover concerns of properly designing a database to suit the task, such as database normalization. As a result, this course is good for setting someone up to use an existing database, but not enough to help them set up a new database. Or at least, not an efficient or effective one. Maybe that’ll be part of another course.

Heroku Free Rides are Over

The Ruby on Rails Tutorial taught me about Heroku, a service for hosting web applications. Your app is running on Amazon Web Services, but Heroku handles all the management and administration of those machines. We just have to focus on our code. It was recommended by the Ruby on Rails Tutorial because it made hosting Rails apps on live internet servers super easy, so we could focus on learning Ruby on Rails and not on AWS administration. And it didn’t cost us anything at the time, as we were able to run on Heroku’s lowest performance free tier.

But that free tier disappears on November 28th, 2022. This is disappointing but not a surprise after Heroku was acquired by Salesforce. Many startups have generous free trials to build up a customer base and prove demand for the product exists. With this proven demand, those startups are acquired by new owners who demand they get serious about making profits. This happened to Cloud 9 IDE, which I played with back when it was free, but that free tier disappeared after it was acquired Amazon. It happened to Ruby on Rails tutorial, which was free but its parent organization Learn Enough Society has since been acquired and everything is now behind a paywall. This, by the way, is one of the reasons I’ve stopped working on micro Sawppy. I had been building my open source Sawppy rover in Onshape because there was a free tier, but they’ve been making it harder and harder to find. After Onshape was acquired, I knew it was only a matter of time before the free tier disappears entirely. I’m looking for another accessible CAD solution for future Sawppy, but that’s a topic for another day.

Today I’m sad at the fact Heroku free tier is going away. I had been studiously learning about HTML, CSS, JavaScript, and just finished a course for Node.js web apps with Express. I’ll definitely start with projects that are deployed only to my home network, but I had grandiose dreams of deploying internet-facing apps via Heroku. I might still do so, but I wouldn’t be able to do it for free. After Heroku announced that free tiers were disappearing, they offered a new tier more affordable than what they had otherwise offered but still not free. Based on the pricing estimator, it looks like a hosted web app equivalent to the previous free tier performance will cost at least $5/month. And if we want a database behind that app, it’ll be another $5/month. Not exactly an extortion, but a significant friction for hobbyists like myself. Maybe they’ll make further adjustments to their pricing structure in the future, I could only hope. In the meantime, I should return to my study.

Replace node-static with serve-static for ESP32 Sawppy Development

One of the optional middleware modules maintained by the Expressjs team is express.static, which can handle serving static assets like HTML, CSS, and images. It was used in code examples for Codecademy’s Learn Express course, and I made a mental note to investigate further after class. I thought it might help me with a problem I already had on hand, and it did!

When I started writing code for a micro Sawppy rover running on an ESP32, I wanted to be able to iterate on client-side code without having to reflash an ESP32. So as an educational test run of Node.js, I wrote a JavaScript counterpart to the code I wrote (in C/C++) for running on ESP32. While they are two different codebases, I intended for the HTTP interface to be identical and indistinguishable by the HTML/CSS/JavaScript client code I wrote. Most of this server-side work was focused around websocket, but I also needed to serve some static files. I looked on nodejs.org and found “How to serve static files” in their knowledge base. That page gave an example using the node-static module, which I copied for my project.

Things were fine for a while, but then I started getting messages from the Github Dependabot nagging me to fix a critical security flaw in my repository due to its use of a library called minimist. It was an indirect dependency I picked up by using node-static, so I figured it’ll be fixed after I pick up an update to node-static. But that update never came. As of this writing, the node-static package on NPM hadn’t been updated for four years. I see updates made on the GitHub repository, but for whatever reason NPM hasn’t picked that up and thus its registry remains outdated.

The good news is that my code isn’t deployed on an internet-facing server. This Node.js server is only for local development of my ESP32 Sawppy client-side browser code, which vastly minimizes the window of vulnerability. But still, I don’t like running known vulnerable code, even if it is only accessible from my own computer and only while I’m working on ESP32 Sawppy code. I want to get this fixed somehow.

After I got nginx set up as a local web server, I thought maybe I could switch to using nginx to serve these static files too. But there’s a hitch: a websocket connection starts as a HTTP request for an upgrade to websocket. So the HTTP server must interoperate with the websocket server for a smooth handover. It’s possible to set this up with nginx, but the instructions to do so is above my current nginx skill level. To keep this simple I need to stay within Node.js.

Which brought me back to Express and its express.static file server. I thought maybe I could fire up an Express app, use just this express.static middleware, and almost nothing else of Express. It’s overkill but it’s not stupid if it works. I just had to figure out how it would handover to my websocket code. Reading Express documentation for express.static, I saw it was built on top of a NPM module called serve-static, and was delighted to learn it can be used independent of Express! Their README included an example: Serve files with vanilla node.js http server and this was exactly what I needed. By using the same Node.js http module, my websocket upgrade handover code will work in exactly the same way. At the end, switching from node-static to serve-static was nearly a direct replacement requiring minimal code edit. And after removing node-static from my package.json, GitHub dependabot was happy and closed out my security issue. I will be free from nagging messages, at least until the next security concern. That might be serious if I had deployed to be internet accessible, but the odds of that just dropped.

Notes on Express “Getting Started” Guide

During the instruction of Codecademy’s “Learn Express” course, we see a few middleware modules that we can optionally use in our project as needed. Examples used in the course are morgan and body-parser, and one of the quizzes asked us to look up vhost. Course material even started using serve-static before we learned about middleware modules at all. These four middleware modules were among those popular enough to be adopted by the Expressjs team who now maintain them.

Since that meant I already had a browser tab open to the Express project site, I decided to poke around. Specifically, I wanted to see how their own Getting Started guide compared to the Codecademy course I just finished. My verdict: the official Express site provides a wider breadth of information but not nearly as much depth for educating a newcomer. If I hadn’t taken the Codecademy course and tried to get started with this site, I would have been able to get a simple Express application up and running but I would not have understood much of what was going on. Especially if I had created an app using the boilerplate application generator. Even after the Codecademy course I don’t know what most of these settings mean!

But the official site had wider breadth, as Codecademy didn’t even mention the boilerplate tool. It also has many lists of pointers to resources, like the aforementioned list of popular middleware modules. Another list I expect to be useful is a sample of options for database integration. Some minimal contextual information was provided with each listed link, but it’s up to us to follow those links and go from there. The only place where this site goes in depth is the Express API reference, which makes sense as the official site for Express should naturally serve as the authoritative source for such information!

I anticipate that I will use Express for at least a few learning/toy projects in the future, at which point I will need to return to this site for API reference and pointers to resources that might help me solve problems in the future. However, before I even get very far into Express, this site has already helped me solve an immediate problem: node-static is out of date.

Notes on Codecademy “Learn Express”

I may have my quibbles with Codecademy’s Learn Node.js course, but it at least gave me a better understanding to supplement what I had learned bumping around on my own. But the power of Node isn’t just the runtime, it’s the software ecosystem which has grown up around it. I have many many choices of what to learn from this point, and I decided to try the Learn Express course.

Before I started the course, I understood Express was one of the earlier Node.js frameworks for building back end of websites in JavaScript. And while there have been many others that have come online since, with more features and capabilities, Express is still popular because it set out not to pack itself with features and capabilities. This meant if we wanted to prototype something slightly off the beaten path, Express would not get in our way. This sounded like a good tool to have in the toolbox.

After taking the course, I learned how Express accomplishes those goals. Express Routes helps us map HTTP methods (GET/POST/PUT/DELETE) to JavaScript code via “Routes”, and for each route we can compose multiple JavaScript modules in the form of “Middleware”. With this mechanism we can assemble arbitrary web API by chaining middleware modules like LEGO pieces to respond to HTTP methods. And… that’s basically the entirety of core Express. Everything else is optional, so we only need to pull in what we need (in the form of middlware modules) for a particular project.

When introducing Routes in Express, our little learning JavaScript handler functions are actually fully qualified Middleware, but we didn’t know it yet. What I did notice is that it had the signature of three parameters: (request, response, next). The Routes course talked about reading request to build our response, but it never talked about next. Students who are curious about them and striking out to search on their own as I did would find information about “chaining”, but it wouldn’t make sense until we learned Middleware. I thought it would have been nice if the course would say “we’ll learn about next later, when we learn about Middleware” or something to that effect.

My gripe with this course is in its quiz sections. We are given partial chunk of JavaScript and told to fill in certain things. When we click “Check Work” we trigger some validation code to see if we did it right. If we did it wrong, we might get an error message to help us on our way. But sometimes the only feedback we receive is that our answer is incorrect, with no further feedback. Unlike earlier Node course exercises, we were not given a command prompt to run “node app.js” and see our output. This meant we could not see the test input, we could not see our program’s behavior, and we could not debug with console.log(). I tried to spin up my own Node.js Docker container to try running the sample code, but we weren’t given entire programs to run and we weren’t given the test input so that was a bust.

I eventually found a workaround: use exceptions. Instead of console.log('debug message') I could use throw Error('debug message') and that would show up on the Codecademy UI. This is far less than ideal.

Once I got past the Route section, I proceeded to Middleware. Most of this unit was focused on showing us how various Middleware mechanisms allow us to reduce code duplication. My gripe with this section is that the course made us do useless repetitive work before telling us to replace them with much more elegant Middleware modules. I understand this is how the course author chose to make their point, but I’m grumpy at useless make-work that I would delete a few minutes later.

By the end of the course, we know Express basics of Route and Middleware and got a little bit of practice building routes from freely available middleware modules. The course ends by telling us there are a lot of Express middleware out there. I decided to look into Express documentation for some starting points.

Ubuntu Phased Package Update

I’m old enough to remember a time when it was a point of pride when a computer system can stay online for long periods of time (sometimes years) without crashing. It was regarded as one of the differentiations between desktop and server-class hardware to justify their significant price gap. Nowadays, a computer with years-long uptime is considered a liability: it certainly has not been updated with the latest security patches. Microsoft has a regular Patch Tuesday to roll out fixes, Apple rolls out their fixes on a less regular schedule, and Linux distributions are constantly releasing updates. For my computers running Ubuntu, running “sudo apt update” followed by “sudo apt upgrade” then “sudo reboot” is a regular maintenance task.

Recently (within the past few months) I started noticing a new behavior in my Ubuntu 22.04 installations: “sudo apt upgrade” no longer automatically installs all available updates, with a subset listed as “The following packages have been kept back”. I first saw this message before, and at that time it meant there were version conflicts somewhere in the system. This was a recurring headache with Nvidia drivers in past years, but that has been most resolved. Also, if this were caused by conflicts, explicitly upgrading the package would list its dependencies. But when I explicitly upgrade a kept-back package, it installed without further complaint. What’s going on?

$ sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
Try Ubuntu Pro beta with a free personal subscription on up to 5 machines.
Learn more at https://ubuntu.com/pro
The following packages have been kept back:
  distro-info-data gnome-shell gnome-shell-common tzdata
The following packages will be upgraded:
  gir1.2-mutter-10 libmutter-10-0 libntfs-3g89 libpython3.10 libpython3.10-minimal libpython3.10-stdlib mutter-common ntfs-3g python3.10 python3.10-minimal
10 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
7 standard LTS security updates
Need to get 1,519 kB/9,444 kB of archives.
After this operation, 5,120 B disk space will be freed.
Do you want to continue? [Y/n]

A web search on “The following packages have been kept back” found lots of ways this message might come up. Some old problems going way back. But since this symptom may be caused by a large number of different causes, we can’t just blindly try every possible fix. We also need some way to validate the cause so we can apply the right fix. I found several different potential causes, and none of the validations applied, so I kept looking until I found this AskUbuntu thread suggesting I am seeing the effect of a phased rollout. In other words: this is not a bug, it is a feature!

When an update is rolled out, sometimes the developers find out too late a problem has escaped their testing. Rolling an update out to everyone at once also means such problems hit everyone at once. Phased update rollout tries to mitigate the damage of such problems: when an update is released, it is only rolled out to a subset of applicable systems. If those rollouts go well, the following phase will distribute the update to more systems, repeating until it is available to everyone. But sometimes somebody wants to skip the wait and install the new thing before their turn in a phased rollout, so they are allowed to “sudo apt upgrade” a package explicitly without error.

So back to the problem validation step: how would we know if a package is kept back due to phased rollout? We can pull up the “apt-cache policy” associated with a package and look for a “phased” percentage associated with the latest version. If so, that means the update is in the middle of a phased rollout. If the updated package is important to us, we can explicitly upgrade now. But if it is not, we can just wait for the phases to include us and be installed in a future “sudo apt upgrade” run.

$ apt-cache policy tzdata
tzdata:
  Installed: 2022e-0ubuntu0.22.04.0
  Candidate: 2022f-0ubuntu0.22.04.0
  Version table:
     2022f-0ubuntu0.22.04.0 500 (phased 10%)
        500 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages
 *** 2022e-0ubuntu0.22.04.0 100
        100 /var/lib/dpkg/status
     2022a-0ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu jammy/main i386 Packages

Conveyer Belt Routing of “Freshly Frosted” Puzzle Game

One of the items on my to-do list is to sit down and gain proficiency in KiCad, the open-source electronic circuit board design software suite. I have had the “Getting Started” guide open in a browser tab for months! I’ve played around with it before to produce simple schematics and board layouts, and I remember routing wires in a schematic/on a board is not an exact science. There’s no one best practice, it’s more of an art that the skilled practitioner can do far better than a beginner. Looking at all the places a wire has to visit, avoiding the places it should not visit, and repeat the process for more wires keeping them out of each other’s way. It tickles a very specific part of my brain.

The same part of my brain came out to play recently in an entirely different context: the single player puzzle game “Freshly Frosted“. A given game board is the floor of a doughnut factory, with all the machines of the assembly line already installed. Our job is to route conveyer belt, so a plain doughnut receives the proper toppings (or none at all) on their way to the delivery counter. Simple in concept, but there is surprising depth. The game has a total of 144 levels, organized by a dozen doughnuts in a dozen boxes. The first level of each box is always very easy, but it introduces a new concept. Each level in the box uses the same concept but increases difficulty. By the time we get to the twelfth and final level of the box, we have a serious challenge on our hands. Fortunately, the game lets us skip levels to a limited degree, so we can set aside some of these challenging puzzles for later.

I felt the levels were accurately sorted by difficulty, which ramps up smoothly. The jump from one level to the next never felt jarring. In my personal experience, most the first box was easily solved within a minute, and I was almost ready to write it off as a simple kid’s game until level 1-11. This eleventh level of the first box was the first one where I had to sit and think over the problem for a bit before I could figure it out. Level 3-12 (final level of the third box) was the first puzzle where I had to leave the game and ponder the problem overnight before I solved it. (Picture of this post.)

My only gripe about this game is a handful of puzzles that didn’t just depend on order of operation, they also depended on the timing of those operations. I would get very close and, to cross the finish line, I had to lengthen or shorten certain belts to adjust timing. I did not enjoy those puzzles, because I end up spending a lot of time struggling to fine tune timing instead of the more enjoyable and rewarding adventure of solving to fit logical constraints. But this fits with my KiCad analogy, where we’d have to struggle with keeping lengths of differential signal wire pairs the same. Routing and timing, just like circuit design!

I wouldn’t go as far as to call Freshly Frosted “PCB Routing: The Game” but it’s pretty darned close. A skilled Freshly Frosted player may or may not find it easier to learn KiCad routing, but I expect it to exercise relevant portions of the brain either way. For the month of November 2022, Freshly Frosted is free for Amazon Prime members to play on Amazon’s Luna game streaming service. Otherwise, it is ~$10 on all the supported game platform stores. Steam for PC, Microsoft Xbox, etc. Highly recommended from me!

Notes on Codecademy “Learn Node.js”

I’ve taken most of Codecademy’s HTML/CSS course catalog for front-end web development, ending with a series of very educational exercises created outside of Codecademy’s learning environment. I think I’m pretty well set up to execute web browser client-side portions of my project ideas, but I also need to get some education on server-side coding before I can put all the pieces together. I’ve played with Node.js earlier, but I’m far from an expert. It should be helpful to get a more formalized introduction via Codecademy, starting with Learn Node.js.

This course recommends going through Introduction to JavaScript as a prerequisite, so the course assumes we already know those basics. The course does not place the same requirement on Intermediate JavaScript, so some of the relevant course material is pulled into this Node.js course. Section on Node modules were reruns for me, but here it’s augmented with additional details and a pointer to official documentation.

The good news for the overlap portions is that it meant I already had partial credit for Learn Node.js as soon as I started, the bad news is the Codecademy’s own back-end got a little confused. I clicked through “Next” for a quick review, and by doing so it skipped me over a few lessons that I had not yet seen. My first hint something was wrong was getting tossed into a progress checking quiz and being baffled: “I don’t remember seeing this material before!” I went back to examine the course syllabus, where I saw the skipped portions. The quiz was much easier once I went through that material!

This course taught me about error-first callback functions, something that is apparently an old convention for asynchronous JavaScript (or just Node) code that I hadn’t been aware of. I think I stumbled across this in my earlier experiments and struggled to use the effectively. Here I learn they were the conceptual predecessor to promises, which led to async/await which plays nice with promises. But what about even older error-first callback code? This is where util.promisify() comes into the picture, so that everyone can work together. Recognizing what error-first callbacks are and knowing how to interoperate via util.promisify(), should be very useful.

The course instructs us on how to install Node.js locally on our development computers, but I’m going to stick with using Docker containers. Doing so would be inconvenient if I wanted to rely on globally installed libraries, but I want to avoid global installations as much as possible anyway. NPM is perfectly happy to work at project scope and that just takes mapping my project directory as a volume into the Docker container.

After all, I did that as a Docker & Node test run with ESP32 Sawppy’s web interface. But that brought in some NPM headaches: I was perpetually triggering GitHub dependabot warnings about security vulnerabilities in NPM modules I hadn’t even realized I was using. Doing a straight “update to latest” did not resolve these issues, I eventually figured out it was because I had been using node-static to serve static pages in my projects. But the node-static package hadn’t been updated in years and so it certainly wouldn’t have picked up security fixes. Perhaps I could switch it to another static server NPM module like http-server, or get rid of that altogether and keep using nginx as sheer overkill static web server.

Before I decide, though, this Learn Node.js course ended with a few exercises building our own HTTP server using Node libraries. These were a little more challenging than typical Codecademy in-course exercises. One factor is that the instructions told us to do a lot of things with no way to incrementally test them as we go. We didn’t fire it up the server to listen for traffic (server.listen()) until the second-from-final step, and by then I had accumulated a lot of mistakes that took time to untangle from the rest of the code. The second factor is that the instructions were more vague than usual. Some Codecademy exercises tell us exactly what to type and on which line, and I think that didn’t leave enough room for us to figure things out for ourselves and learn. This exercise would sometimes tell us “fill in the request header” without details or even which Node.js API to use. We had to figure it all out ourselves. I realize this is a delicate balance when writing course material. I feel Codecademy is usually too much “do exactly this” for my taste, but the final project of Learn Node.js might have gone too far in the “left us flailing uselessly” direction.

In the meantime, I believe I have enough of a start to continue learning about server-side JavaScript. My next step is to learn Express.

Notes on Codecademy “Build a Website” Off-Platform Projects

Most Codecademy courses involve interactive learning inside their in-browser learning development environment, but occasionally we are directed to get off Codecademy platform and build something on our own. I have set up nginx as a local development web host (not the best use of nginx) serving files directly off a GitHub repository for these projects. This repository is, in turn, set up to host project content via GitHub pages. After this infrastructure is setup, I dove in to the off-platform project assignments of Built a Website with HTML, CSS, and Github Pages skill path.

The first project was “Dasmoto’s Arts & Crafts”, a relatively simple art shop landing page exercising a beginner’s level of HTML and CSS. We are given the images to use, and a specification of how the site should look. This was a practice exercise intended for us to run directly off local filesystem, without even a web server. But where’s the fun in that? I built this project locally, serving my files via nginx.

The next project was “Tea Cozy”, a more sophisticated tea shop landing page. This was from “Flexbox and Grid” section that pulled in most of the material of Learn CSS: Flexbox and Grid. Again, we are given a set of images to use, and a specification for how the site should look. This layout is far more complex than “Dasmoto’s Arts & Crafts” project, requiring use of (no surprise) flexbox and grid. I enjoyed the challenge of building “Tea Cozy” and I feel I have a much better grasp of flexbox & grid after this project.

Towards the end of the skill path was a project “Excursion”, a coming-soon phone app landing page. In addition to the images, we also had a video to embed. I had thought it be more of a skill practice than “Tea Cozy”, but it turned out to be far simpler with minimal layout challenges. The focus of this exercise was on GitHub Pages, a topic I had already put in the time to learn, so I blitzed through it relatively quickly. My only problem was trying to incorporate the copyright symbol, which wasn’t as simple as copy and pasting the Unicode character. A strange character gets added whenever I try to do so! I decided this problem wasn’t technically a HTML/CSS issue and punted.

And finally, we have a capstone project “Colmar Academy” educational institution landing page. We have a lot of added complexity in this project. This is the first project to require responsive layout, with both desktop and mobile views required. Some of the images provided had corresponding high-resolution desktop and low-resolution mobile versions. There was a video, and we even get a few icons in the form of SVG files. The specification we were given for this project was more loosely defined, with fewer explicit details, and we are to use our design sense to fill in the gaps. For example, it was up to us to decide where our media query breakpoints would be to transition between desktop and mobile views. This project took a lot of time, but it was time well spent because of everything I learned while doing it. At the moment, my biggest unsolved mystery is how to switch between desktop and mobile images from CSS. I couldn’t change the value of src property on an <img> tag from CSS! I ended up using two <img> tags, one with the desktop image and one with the mobile image and using CSS media query to set one of them to display: none; This feels inelegant, and I hope I learn a better way to do this in the future.


My code for these assignments are publicly visible on GitHub.

Local Development Web Host nginx Docker Container

During the course of Codecademy’s skill path for website publishing, we are given several off-platform assignments. “Off-platform” in this context means we are to build a website on our own using something outside Codecademy’s in-browser learning environment. I decided to put these assignments on my GitHub account, because then it’s easy to publish them via GitHub Pages. (When I decided this, I hadn’t realized GitHub Pages would be the explicit focus for one of the assignments.) But there’s a several-minute delay between pushing git commits and seeing those changes reflected on GitHub Pages. So I also wanted a local development web host for immediate feedback as I work. I decided to try using nginx for this purpose.

Local development web hosting is just about the lightest duty workload possible for web server software, so using nginx is sheer overkill. The renowned speed and response of nginx running high traffic websites is completely wasted serving my single browser. Furthermore, some of nginx performance is due to its high-performance caching system, and I wanted to turn that off as well. Running nginx and not caring about cache is like buying a Toyota Prius and not caring about fuel efficiency. Such is the contradiction of using nginx as a local development web host. I will be making many changes and I want to see their effect immediately. I don’t want to risk looking at results from a stale cached copy.

The reason I’m using this overkill solution (or arguably the wrong tool for the job) is because I hoped it would give me a beginner’s level view of working with nginx. The easy part comes from the fact nginx distributes their code as a Docker container, so I could quickly launch an instance of “nginx:stable-alpine” and play with it. According to the tagging schema described on nginx Docker Hub page, “stable” represents the latest stable release which is fine by me as I don’t need the latest features. And “alpine” refers to a container built on top of Alpine distribution of Linux with a focus on minimal size and complexity.

To disable caching, I copied the default configuration file (/etc/nginx/nginx.conf) out of the nginx container so I could add a few lines to the configuration file. One to turn off nginx server side caching (from nginx documentation) and another to ask browser not to cache (from this StackOverflow post.)

    # Ask server not to cache
    proxy_no_cache $http_pragma $http_authorization;

    # Ask browser not to cache
    add_header 'Cache-Control' 'no-cache, no-store, must-revalidate';

After editing, I will use Docker to map my modified version overriding the default. I don’t think this is best Docker practice, but I’m focused on “easy” right now. I think the “right” way to do this is to build my own docker container on top of the nginx release but after modifying its configuration file. Something like what’s described in this person’s blog post.

I specified the settings I typically use in a Docker Compose file. Now all I need to do is to go into my project directory and run “docker compose up” to have a non-caching local development web host. To double check, I used curl -I (uppercase i) to verify my intended Cache-Control headers have been added.

$ curl -I http://localhost:18080
HTTP/1.1 200 OK
Server: nginx/1.22.1
Date: Wed, 09 Nov 2022 20:07:11 GMT
Content-Type: text/html
Content-Length: 4868
Last-Modified: Wed, 09 Nov 2022 18:28:46 GMT
Connection: keep-alive
ETag: "636bf15e-1304"
Cache-Control: no-cache, no-store, must-revalidate
Accept-Ranges: bytes

Looks good! My modified nginx.conf and my docker-compose.yml files are publicly available from my Codecademy HTML off-platform projects repository. After this work getting nginx set up for local hosting and GitHub Pages set up for global hosting, it’s time to jump into my Codecademy off-platform assignments!

Notes on Codecademy “Build a Website with HTML, CSS, and Github Pages” Skill Path

After finishing Codecademy’s navigation design course, I thought it had some interesting information but it also spent too much time on CSS tricks I did not expect to be broadly applicable to future projects. Completing that course also meant I had covered majority of Codecademy’s courses under HTML & CSS section of the catalog. However, there are a few items listed that were not “Courses” so I thought I would check out a “Skill Path”. According to Codecademy, a skill path is focused on delivering the knowledge necessary to accomplish certain tasks. I paraphrase it as “Teach me what I need to know to accomplish X” versus a course which is more “Tell me about Y and how I might use it.”

In practice, judging by my first skill path “Build a Website with HTML, CSS, and Github Pages” (Or the shorter “Learn How to Build Websites” as per the URL) a skill path repackages a lot of components pulled from other Codecademy resources. Mostly individual lessons (modules?) but also other resources like their articles and blogs. After taking majority of Codecademy courses on HTML/CSS, going through this skill path was a little disorienting because their backend had tracked which modules I’ve already done. This meant that as soon as I clicked on starting this skill path, my progress was immediately over 50% complete. Looking over the skill path syllabus, I could see what I’ve already done and the gaps I still need to cover.

Most of the gaps were information presented Codecademy articles, covering things like how to set up a code editor like Visual Studio Code. (My personal choice.) Some of the gaps were modules on courses I hadn’t bothered to take, for example the command line course as I was already quite comfortable, but I was able to blitz through quickly.

A surprise was the gap on web accessibility. I thought this was an error as I had taken their Learn CSS Accessibility course, but the database is correct: this was a different course with material I had wished was in the CSS accessibility course. Starting with basic background and on to how to set up a screen reader for us to explore how these features will be consumed. I also appreciated more information on ARIA roles, where I learned we can put down some very fine-grained annotations for accessibility. There are a lot more ARIA roles than there are semantic HTML elements. It’ll take a lot of learning and practice to do ARIA well, but if the spec is too overwhelming, we can start with MDN’s introduction to ARIA.

I was heartened by this coverage of web accessibility but was then disappointed by its coverage of Font Awesome. Which I learned is a huge collection of icons (apparently not fonts as the name implies) available for use in websites. Icons are inherently compact way of visual communication, so we need to pay more attention to their use to ensure they are accessible. Unfortunately, not only did the course not cover how to maintain accessibility, it does not even mention accessibility as a concern when using icons.

One section I’m glad they put in this course is “Documentation and Research”. There’s no way for the course to cover everything, so it needs to teach people how to look stuff up on their own. For web developers, this means the holy trinity of MDN, Google, and StackOverflow. And for beginners who needed the exercise, a broken web site to fix by looking up the problems.

The real star of the skill path, though, are the off-platform projects. I like learning with Codecademy and its embedded interactive development environment. We can get a lesson side by side with sample code we can play with. However, these are all fairly basic fill-in-the-blank types of exercises. To be a web developer we need to be able to build a page from scratch, which is where these off-platform projects come in. We are given the assets (images and occasionally video) and a specification of what to build, but no templates. We had to create our own index.html and style.css from scratch and serve it up to in a browser to see our results. This course covered developing on the local file system, and using GitHub Pages, but I decided to add one more option to the mix: I thought it’d be a good exercise to setup nginx for local development hosting.

Notes on Codecademy “Learn Navigation Design”

I was definitely out of my depth with Codecademy’s color design course, but I was happy to absorb what I can and move on to another topic of novelty: Codecademy’s “Learn Navigation Design” course. Just as color could give subtle hints to the user on how to best interact with the site, so does applying good design to navigation elements. It’s something that we would rarely consciously notice until we encounter a poorly designed page, which is of course how this course started: by showing us an intentionally badly designed page and go up from there.

I was surprised that the first topic was how to show links on a page. After all the link styling in previous CSS courses and speaking of the user agent (browser default) stylesheet as a source of problems, this course presents the other side of the story: Hang on, guys, there are good reason they’re the way they are! And if we arbitrarily toss out all of those traits, site usability will suffer. Hover states are discussed here, and this time we’re reminded of their absence on touchscreen devices. We also get a link to MDN on pseudo-classes, information missing from the color design course!

Moving on from links to buttons, it started with an explanation of skeuomorphism vs. flat design for user interactive elements like buttons. This course covers examples for both styles. I’m personally a fan of the flat school of design. If somebody wants to do skeuomorphism on a button, I demand that they look like keys on an IBM Selectric typewriter.

After buttons the course talked about secondary navigation in the form of breadcrumbs on the page, usually found at the top of a site just before the header block. I appreciate an overview of the concept, but some of the examples get into fancy CSS tricks. I don’t think they’ll be generally applicable to all sites and I’m wary they degrade a page’s accessibility.

This navigation design course barely scratched the surface of User Experience (UX) design, but of course there’s an entirely separate Codecademy course “Introduction to UI and UX Design“. Looking over its syllabus, it doesn’t feel like the material would be useful in my personal tinkering projects. There’s also the fact that course was “Built in partnership with Figma” and the final section of the course is “Prototyping with Figma.” Is this course just an extended ad for Figma? I don’t know and at the moment I’m not terribly interested in finding out. At least Figma offers a free starter tier, if I decide to come back to this later.

Right now, I’m more curious about checking out Codecademy’s “Skill Path” offerings.

Notes on Codecademy “Learn Color Design”

After a brief overview of CSS browser compatibility concerns, which wrapped up Codecademy’s Learn Intermediate CSS curriculum, I looked at what remained under Codecademy’s HTML/CSS umbrella and started their Learn Color Design course. This is a less technical course more focused on art & design perspectives, so I knew it would take more effort for me to grasp all the concepts.

At least they started easy (for me) by going over HSL versus RGB for specifying color, and how HSL is easier to work with from a color design perspective. This is something I had learned from working with Pixelblaze and most of the concepts transfer easily. But then we moved quickly into concepts I had never encountered before, like designing color schemes. I liked the fact that the course material stayed with a same example page and changed colors around to illustrate monochromatic, complementary, analogous, and triadic color schemes. Keeping the content identical and changing just the color did help me see the effect somewhat, even if I am not familiar with the kind of vocabulary used. For example, I don’t know what it means for a color scheme to “create a sense of equality, vibrancy, and security in your designs”. These “Color Psychology” concepts are very foreign to my brain and will take time to absorb. Some of the vocabulary is new to me, too, using familiar words in unfamiliar ways. There was a quiz question “How is a shade of a color produced” and none of the possible answers made sense to my brain until I returned to course material and reviewed how the vocabulary is defined in this specific context.

This course had a gem of a quote that I wished more web designers took to heart:

Remember that most users skim websites! They are not reading every word and checking every menu—you need to guide the user to the most important content with good color choices.

Like the lecture, the practice exercise gave us a site that was mostly grayscale and had us add color to it. The instruction ends with an encouraging “Now our site is looking great!” but it really doesn’t. I have yet to master the subtleties in choosing colors and to compensate I intend to use color schemes built by others as much as I could. But this course gave me some foundation so I could appreciate those prebuilt color schemes. It also helped me appreciate BrandColors, a collection of color schemes associated with many brands we see in our everyday lives.

There was an optional resource that pointed to Adobe Color with the claim “you will learn to use Adobe Color” but when clicking the link, I was redirected straight to the color wheel tool. It’s not immediately obvious where I could find anything instructional to help me learn, I think that URL might be outdated. The same “dropped into a tool with no instructions” problem applied to another optional resource, a color tool by CloudFlare Design. (Not to be confused with the CloudFlare that handles DDoS attacks.)

After some experimentation with color, we are to put that theory into practice by applying color for UI. Again, the instruction materials used a sample page that started out mostly grayscale and we added colors as we go. UI-specific concepts are added, such as using colors for button hover and disabled states. An aside: I wish this class discussed the fact that hover is absent from touchscreens and how design should change in response. And speaking of wishes, an earlier wish was granted here: we finally have a discussion on color blindness and given ColorSafe as a tool to help. Another realization I had during this course is that we never really had a discussion on CSS pseudo-classes, which we use to style things like hover states. A quick web search found this MDN resource as a starting point for later research. For now, I will proceed to the navigation design course.