Notes on Reinforcement Learning Algorithm Implementations Published by OpenAI

OpenAI’s Spinning Up in Deep RL guide has copious written resources, but it also offers resources in the form of algorithm implementations in code. Each implementation is accompanied by a background section talking about how the algorithm works, summarized by a “Quick Facts” section that serves as a high-level view of how these algorithms differ from each other. As of this writing, there are six implementations. I understand the main division is that there are three “on-policy” algorithms and three “off-policy” algorithms. Within each division, the three algorithms are roughly sorted by age and illustrate an arc of researchin the field. The best(?) on-policy algorithm here is Proximal Policy Optimization (PPO) and representing off-policy vanguard is Soft Actor-Critic (SAC).

These implementations are geared towards teaching these algorithms. Since priority is placed on this learning context, these implementations are missing enhancements and optimizations that would make the code harder to understand. For example, many of these implementations do not support parallelization.

This is great for its intended purpose of supplementing Spinning Up, but some people wanted an already optimized implementation. For this audience, OpenAI published the OpenAI baselines a few years ago to serve as high water marks for algorithm implementation. These baselines can serve either as starting point for other projects, or benchmarks to measure potential improvements against.

However, it appears these baselines have gone a little stale. Not any fault of the implementations themselves, but merely due to the rapidly moving nature of this research field. The repository hosts implementations using TensorFlow version 1.0, which has been deprecated. There’s a (partial? full?) conversion to TensorFlow 2.0 in a separate branch, but that never merged back to the main branch for whatever reason. As of this writing there’s an open issue asking for PyTorch implementations, prompted by OpenAI’s own proclamation that they will be standardizing on PyTorch. (This proclamation actually led to the PyTorch conversion of Spinning Up in Deep RL examples.) However, there’s no word yet on PyTorch conversions for OpenAI baselines so anyone who want implementations in PyTorch will either have to optimize the Spinning Up implementations themselves or look elsewhere.

Of course, all of this assumes reinforcement learning will actually solve the problem, which I’ve learned might not be a good assumption.

Notes on Deep Reinforcement Learning Resources by OpenAI

I’m going through the Spinning Up in Deep RL (Reinforcement Learning) guide published by OpenAI, and I found the Introduction to RL section quite enlightening. While it was just a bit too advanced for me to understand everything, I understood enough to make sense of many other things I’ve come across recently as I had browsed the publicly available resources in this field. Following that introductory section was a Resources section with the following pages:

Spinning Up as a Deep RL Researcher

This section is for a target audience that wishes to contribute to the frontier of research in reinforcement learning. Building the background, learn by doing, and how to be rigorous in research projects. This section is filled with information that feels like it was distilled from the author’s experience. The most interesting part for me is the observation that “broken RL code almost always fails silently“. Meaning there are no error messages, just a failure for the agent to learn from its experience. The worst part when this happens is that it’s hard to tell the difference between a flawed algorithm and a flawed implementation of a good algorithm.

Key Papers in Deep RL

This was the completely expected directory of papers, roughly organized by the taxonomy used for the introduction in this guide. I believe the author assembled this list to form a solid foundation for further exploration. It appears that most (and possibly all) of them are freely available, a refreshing change from the paywalls that usually block curious minds.

Exercises

Oh no, I didn’t know there would be homework in this class! Yet here we are. A few problems for readers to attempt implementing using their choice of TensorFlow or PyTorch, along with solutions to check answers against. The first set cover basic implementation, the second set covers some algorithm failure modes. This reinforces what was covered earlier: broken RL code fails silently, so an aspiring practitioner must recognize symptoms of failure modes.

Benchmarks for Spinning Up Implementations

The Spinning Up in Deep RL guide is accompanied by implementations of several representative algorithms. Most of them have two implementations: one in TensorFlow and one in PyTorch. But when I run them on my own computer, how would I know if they’re running correctly? This page is a valuable resource to check against. It has charts for these algorithms’ performance in five MuJoCo Gym environments under conditions also described on this page. And once I have confidence they are running correctly, these are also benchmarks to see if I can improve their performance. For these implementations are written for beginners to read and understand, which means skipping some of the esoteric hard-to-understand optimizations that readers are challenged to put in themselves.

For people who want optimized implementations of these algorithms OpenAI has them covered too. Or at least, they used to.

Notes on “Introduction to RL” by OpenAI

With some confidence that I could practice simple reinforcement learning algorithms, I moved on to what I consider the meat of OpenAI’s Spinning Up in Deep RL guide: the section titled Introduction to RL. I thought it was pretty good stuff.

Part 1: Key Concepts in RL

Since I had been marginally interested in the field, the first section What Can RL Do? was mostly review for me. More than anything else, it confirmed that I was reading the right page. This section proceeded to Key Concepts and Terminology and that quickly caught up then surpassed what knowledge I already had. I was fine with the verbal descriptions and the code snippets, but once the author started phrasing in terms of equations it became more abstract than what I can easily follow. I’m a code person, not an equations person. Given my difficulty comprehending the formalized equations, I was a little bemused at the third section (Optional) Formalism as things were already becoming more formalized than my usual style.

Part 2: Kinds of RL Algorithms

Given that I didn’t comprehend everything in part 1, I was a little scared at what I would find in part 2. Gold. I found pure gold. Here I found a quick survey of major characteristics of various reinforcement learning approaches, explaining those terms in ways I (mostly) understood. Explanations like what “Model-Free” vs. “Model-Based” means. What “on-policy” vs. “off-policy” means. I had seen a lot of these terms thrown around in things I read before, but I never found a place laying them down against one another until I found this section. This section alone was worth the effort of digging into this guide and will help me understand other things. For example, a lot of these terms were in the documentation for Unity ML-Agents. I didn’t understand what they meant at the time, but now I hope to understand more when I review them.

Part 3: Intro to Policy Optimization

Cheered by how informative I found part 2, I proceeded to part 3 and was promptly brought back down to size by the equations. The good thing is that this section had both equations and a code example implementation, and they were explained more or less in parallel. Someone like myself can read the code and try to understand the equations. I expect there are people out there who are the reverse, more comfortable with the equations and appreciate seeing what they look like in code. That said, I can’t claim I completely understand the code, either. Some of the mathematical complexities represented in the equations were not in the sample source code: They were handed off to implementations in PyTorch library.

I foresee a lot of studying PyTorch documentation before I can get proficient at writing something new on my own. But just from reading this Introduction to RL section, I have the basic understanding to navigate PyTorch API documentation and maybe the skill to apply existing implementations written by somebody else. And as expected of an introduction, there are tons of links to more resources.

Old PyTorch Without GPU Is Enough To Start

I’ve mostly successfully followed the installation instructions for OpenAI’s Spinning Up in Deep RL. I am optimistic this particular Anaconda environment (in tandem with the OpenAI guide) will be enough to get me off the ground. However, I don’t expect it to be enough for doing anything extensive. Because when I checked the installation, I saw it pulled down PyTorch 1.3.1 which is now fairly old. As of this writing, PyTorch LTS is 1.8.2 and Stable is 1.10. On top of that, this old PyTorch runs without CUDA GPU acceleration.

>>> print(torch.__version__)
1.3.1
>>> print(torch.cuda.is_available())
False

NVIDIA’s CUDA API has been credited with making the current boom in deep learning possible, because CUDA opened up GPU hardware for usage other than its original gaming intent. Such massively parallel computing hardware made previously impractical algorithms practical. However, such power does incur overhead. For one thing, data has to be copied from a computer’s main memory to memory modules on board the GPU before they can be processed. And then, the results have to be copied back. For smaller problems, the cost of such overhead can swamp the benefit of GPU acceleration.

I believe I ran into this when working through Codecademy’s TensorFlow exercises. I initially set up CPU-only TensorFlow on my computer to get started and, once I had that initial experience, I installed TensorFlow with GPU support. I was a little surprised to see that small teaching examples from Codecademy took more time overall on the GPU accelerated installation than the CPU-only installation. One example took two minutes to train in CPU-only mode, but took two and a half minutes with GPU overhead. By all reports, the GPU will become quite important once I start tackling larger neural networks, but I’m not going to sweat the complexity until then.

So for Spinning Up in Deep RL, my first goal is to get some experience running these algorithms in a teaching examples context. If I am successful through that phase (which is by no means guaranteed) and start going beyond small examples, then I’ll worry about setting up another Anaconda environment with a modern version of PyTorch with GPU support.

Installing Code for OpenAI “Spinning Up in Deep RL”

I want to start playing with training deep reinforcement learning agents, and OpenAI’s “Spinning Up in Deep RL” guide seems like a good place to start. But I was a bit worried about the age of this guide, parts of which has become out of date. One example is that it still says MuJoCo requires a paid license but actually MuJoCo has since become free to use.

Despite this, I decided it’s worth my time to try its software installation guide on my computer running Ubuntu. OpenAI’s recommended procedure uses Anaconda, meaning I’ll have a Python environment dedicated to this adventure and largely isolated from everything else. Starting from Python version (3.6, instead of the latest 3.10) and on down to all the dependent libraries. The good news is that all the Python-based infrastructure seemed to work without problems. But MuJoCo is not Python and thus not under Anaconda isolation, so all my problems came from trying to install mujoco_py. (A Python library to bridge MuJoCo.)

The first problem was apparently expected by the project’s authors, as I got a prompt to set my LD_LIBRARY_PATH environment variable. The script that gave me this prompt even gave me its best guess on the solution:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/roger/.mujoco/mujoco210/bin

That looked reasonable to me, so I tried it and it successfully allowed me to see the second and third problems. Both were Ubuntu packages that were not explicitly named in the instructions, but I had to install them before things could move on. I did a web search for the first error, which returned the suggestion to run “sudo apt install libosmesa6-dev libgl1-mesa-glx libglfw3” and that seemed to have resolved this message:

/home/roger/anaconda3/envs/spinningup/lib/python3.6/site-packages/mujoco_py/gl/osmesashim.c:1:10: fatal error: GL/osmesa.h: No such file or directory

The second missing package error was straightforward to fix, running “sudo apt install patchelf” to resolve this message:

FileNotFoundError: [Errno 2] No such file or directory: 'patchelf': 'patchelf'

The final step was to install dependencies for MuJoCo-based Gym environments. The instruction page said to run pip install gym[mujoco,robotics] but when I did so, I was distressed to see my work to install mujoco-py was removed. (!!)

  Attempting uninstall: mujoco-py
    Found existing installation: mujoco-py 2.1.2.14
    Uninstalling mujoco-py-2.1.2.14:
      Successfully uninstalled mujoco-py-2.1.2.14

The reason pip wanted to do this was because the gym packages said they want mujoco_py version greater than or equal to 1.5 but less than 2.0.

Collecting mujoco-py<2.0,>=1.50
Downloading mujoco-py-1.50.1.68.tar.gz (120 kB)

Fortunately(?) this attempt to install 1.50.1.68 failed and pip rolled everything back, restoring 2.1.2.14 that I had already installed. Leaving me in a limbo state.

Well, when in doubt, try the easy thing first. I decided to be optimistic and moved on to “check that things are working” command to run Walker2d-v2 environment. I saw a lot of numbers and words flashed by. I only marginally understood them, but I didn’t see anything I recognized as error messages. So while I see a few potential problems ahead, right now it appears this mishmash of old and new components will work well enough for a beginner.

Window Shopping: OpenAI Spinning Up in Deep Reinforcement Learning

I was very encouraged after my second look at Unity ML-Agents. When I first looked at it a few years ago, it motivated me to look at the OpenAI Gym for training AI agents via reinforcement learning. But once I finished the “Hello World” examples, I got lost trying to get further and quickly became distracted by other more fruitful project ideas. With this past experience in mind, I set out to find something more instructive for a beginner and found OpenAI’s guide “Spinning Up in Deep RL“. It opened with the following text:

Welcome to Spinning Up in Deep RL! This is an educational resource produced by OpenAI that makes it easier to learn about deep reinforcement learning (deep RL).

This sounds exactly like what I needed. Jackpot! At first I thought this was something new since my last look, but the timeline at the bottom of the page indicated this was already available when I last looked at reinforcement learning resources on OpenAI. I had missed it! I regret the lost opportunity, but at least I’ve found it this time.

The problem with finding such a resource a few years after publication is that it may already be out of date. The field of deep learning moves so fast! I’m pretty sure the fundamentals will still be applicable, but the state of the art has certainly moved on. I’m also worried about the example code that goes with this resource, which looks stale at first glance. For example, it launched with examples that used the now-deprecated TensorFlow 1 API. (Instead of the current TensorFlow 2 API.) I don’t care to learn TF1 just for the sake of this course, but fortunately in January 2020 they added alternative examples implemented using PyTorch instead. If I’m lucky, PyTorch hasn’t made a major version breaking change and I could still use those examples.

In addition to the PyTorch examples, there’s another upside of finding this resource now instead of later. For 3D environment simulations OpenAI uses MuJoCo. When I looked at OpenAI Gyms earlier, running the 3D environments require a MuJoCo license that costs $500/year and I couldn’t justify that money for playing around. But good news! MuJoCo is now free to use.

Window Shopping: Google Machine Learning Crash Course

I’ve learned some machine learning fundamentals through Codecademy Pro, and I took a quick look at what Kaggle offered for free. There’s another option, from another name that (for better and worse) taken leadership in monetizing machine learning: Google. Their freely available Machine Learning Crash Course supposedly started as a way for Google to get their own developers up to speed, and has since been made suitable for public consumption and now available for all.

My favorite part of this course was their prerequisites & prework page. I used this as a guide to choose some of my courses on Codecademy Pro, it’s how I knew Numpy and Pandas were important to the field. However their linked “UltraQuick Tutorial” for those libraries were a bit too terse for me. I have no doubt they provided enough background for people smarter than I am. Personally, I needed help from Codecademy to get ramped up on NumPy and Pandas.

For interactive learning, these courses use Google Colab. Which feels a lot like Kaggle in the sense that they are both cloud-hosted Jupyter (or at least Jupyter style) notebooks. They differ by the server-end hardware, and the software images running on their hardware. I’m too new at this to have any feel of the tradeoffs between those two options, but I do understand the storage difference: Google Colab uses my Google Drive account for storage.

The primary reason I didn’t go with this Machine Learning Crash Course was the fact that instructions came in the form of video lectures, and that turns me off. For self-directed education I prefer the written word because I can go at my own pace and I can easily search and skip around. Many people prefer video lectures as the most direct counterpart to in-person instruction, but I don’t see it that way. For me the most valuable aspect of in-person instruction is the interactive question-and-answer aspect, which is completely lost in a video lecture.

And finally, this course does not cover reinforcement learning, which is my primary interest in applying machine learning. A search for RL on Google developer site leads to the Agents library and associated documentation, but that material was written to help apply reinforcement learning instead of teaching it. I’ll return to the TensorFlow Agents library after getting ramped up on deep RL somewhere else.

Window Shopping: Kaggle Courses

Over the past few months, I’ve spent much of my free time learning with a Codecademy Pro subscription. I was on a self-curated curriculum starting with “Learn Python 3” and ending with “Building Deep Learning Models on TensorFlow”. I felt it was time (and money) well-spent, helping me get up to speed on a lot of concepts and vocabulary on machine learning with deep learning. I enjoyed it a lot and can recommend going through the same path I did, with upsides and caveats documented over the past few blog entries.

But no one path is best for everyone, so I thought it’s worth noting a few alternative approaches. Top of the list is the learning or “Courses” section on Kaggle. They are free of monetary cost and does not require equivalent of a Codecademy Pro subscription. The course materials are Kaggle’s interactive documents that resemble Jupyter notebooks. (Whether Kaggle was inspired by Jupyter or actually cloud-hosted Jupyter I don’t know.) But it provides an interactive learning environment of a different style than Codecademy’s learning environment. In hindsight I can see how this approach might be better. Codecademy course tells us about Jupyter and Kaggle and encourages us to learn them, but going with Kaggle courses means we can start with The Real Deal immediately. Another upside is that I found it much easier to navigate/skip/skim through Kaggle course sections in their notebook format. In comparison I thought it was cumbersome to navigate through Codecademy units whenever I want to go back and review material.

One downside is that these courses assume more background than Codecademy does. For example there’s nothing to teach basic statistics and probability — the student is assumed to know them (or learn them elsewhere.) A final downside for me personally is that there’s very little here on applying deep learning to reinforcement learning. Their “Intro to Game AI and Reinforcement Learning” unit has just a single notebook “Deep Reinforcement Learning” that opens with the disclaimer:

In this notebook, we won’t be able to explore this complex field in detail, but you’ll learn about the big picture and explore code that you can use to train your own agent.

Given that what I find interesting is missing, I don’t think I’ll spend very much time on Kaggle. In principle, there’s value in seeing Kaggle instructors take on the same fundamentals for a more well-rounded foundation. In practice I’m impatient to get closer to my target. Perhaps I’ll return later for a refresher. In the meantime, Kaggle Learn might be a better option than Codecademy Pro for those who (A) already have a stronger background in the area, or (B) who prefer the notebook learning format, or (C) could not justify spending money for Codecademy Pro.

And for those that prefer video lecture format, there’s Google’s Machine Learning Crash Course.

Notes on Codecademy “Build Deep Learning Models with TensorFlow”

Once I upgraded to a Codecademy Pro membership, I started taking courses from its Python catalog with the goal of building a foundation to understand deep learning neural networks. Aside from a few scenic detours, most of my course choices were intended to build upon each other to fulfill what I consider prerequisites for a Codecademy “Skill Path”: Build Deep Learning Models with TensorFlow

This was the first “Skill Path” I took, and I wasn’t quite sure what to expect as Codecademy implied they are different than the courses I took before. But once I got into this “skill path”… it feels pretty much like another course. Just a longer one, with more sessions. It picked up where the “Learn the Basics of Machine Learning” course left off with neural perceptrons, and dived deeper into neural networks.

In contrast to earlier courses that taught various concepts by using them to solve regression problems, this course spent more time on classification problems. We are still using scikit-learn a lot, but as promised by the title we’re also using TensorFlow. Note the course work mostly stayed in the Keras subset of TensorFlow 2 API. Keras used to be a separate library for making it easier to work with TensorFlow version 1, but it has since been merged into TensorFlow version 2 as part of the big revamp between versions.

I want to call attention to an item linked as “additional resources” for the skill path: a book titled “Deep Learning with Python” by François Chollet. (Author, or at least one of the primary people, behind Keras.) Following various links associated with the title, I found that there’s since been a second edition and the first chapter of the book is available to read online for free! I loved reading this chapter, which managed to condense a lot of background on deep learning into a concise history of the field. If the rest of the book is as good as the first chapter, I will learn a lot. The only reason I haven’t bought the book (yet) is that, based on the index, the book doesn’t get into unsupervised reinforcement learning like the type I want to put into my robot projects.

Back to the Codecademy course…. err, skill path: we get a lot of hands-on exercises using Keras to build TensorFlow models and train them on data for various types of problems. This is great, but I felt there was a significant gap in the material. I appreciated learning that different loss functions and optimizers will be used for regression versus classification problems, and we put them to work in their respective domains. But we were merely told which function to use for each exercise, the course doesn’t go into why they were chosen for the problem. I had hoped that the Keras documentation Optimizers Overview page would describe relative strengths and weaknesses of each optimizer, but it was merely a list of optimizers by name. I feel like such a comparison chart must exist somewhere, but it’s not here.

I didn’t quite finish this skill path. I lost motivation to finish the “Portfolio Project” portion of the skill path where we are directed to create a forest cover classification model. My motivation for deep learning lies in reinforcement learning, not classification or regression problems, so my attention has wandered elsewhere. At this point I believe I’ve exhausted all the immediately applicable resources on Codecademy as there are no further deep learning material nor is there anything on reinforcement learning. So I bid a grateful farewell to Codecademy for teaching me many important basics over the past few months and started looking elsewhere.

Notes on Codecademy Intermediate Python Courses

I thought Codecademy’s course “Getting Started Off Platform for Data Science” really deserved more focus than it did when I initially browsed the catalog, regretting that I saw it at the end of my perusal of beginner friendly Python courses. But life moves on. I started going through some intermediate courses with an eye on future studies in machine learning. Here are some notes:

  • Learn Recursion with Python I took purely for fun and curiosity with no expectation of applicability to modern machine learning. In school I learned recursion with Lisp, a language ideally suited for the task. Python wasn’t as good of a fit for the subject, but it was alright. Lisp was also the darling of artificial intelligence research for a while, but I guess the focus has since evolved.
  • Learn Data Visualization with Python gave me more depth on two popular Python graphing libraries: Matplotlib and Seaborn. These are both libraries with lots of functionality so “more depth” is still only a brief overview. Still, I anticipate skills here to be useful in the future and not just in machine learning adventures.
  • Learn Statistics with NumPy was expected to be a direct follow-up to the beginner-friendly Statistics with Python course, but it was not a direct sequel and there’s more overlap than I thought there’d be. This course is shorter, with less coverage on statistics but more about NumPy. After taking the course I think I had parsed the course title as “(Learn Statistics) with NumPy” but I think it’s more accurate to think of it as “Learn (Statistics with NumPy)”
  • Linear Regression in Python is a small but important step up the foothills on the way to climbing the mountain of machine learning. Finding the best line to fit a set of data teaches important concepts like loss functions. And doing it on a 2D plot of points gives us an intuitive grasp of what the process looks like before we start adding variables and increasing the number of dimensions involved. Many concepts are described and we get exercises using the scikit-learn library which implements those algorithms.
  • Learn the Basics of Machine Learning was the obvious follow-up, diving deeper into machine learning fundamentals. All of my old friends are here: Pandas, NumPy, scikit-learn, and more. It’s a huge party of Python libraries! I see this course as a survey of major themes in machine learning, of which neural networks was only a part. It describes a broader context which I believe is a good thing to have in the back of my head. I hope it helps me avoid the trap of trying to use neural nets to solve everything a.k.a. “When I get a shiny new hammer everything looks like a nail”.

Several months after I started reorienting myself with Python 3, I felt like I had the foundation I needed to start digging into the current state of the art of deep learning research. I have no illusions about being able to contribute anything, I’m just trying to learn enough to apply what I can read in papers. My next step is to learn to build a deep learning model.

Notes on Codecademy “Getting Started Off Platform for Data Science”

I like Codecademy’s format of having a bit of information that is followed immediately by an opportunity to try it myself. I like learn-by-doing as a beginner, even if the teaching/learning environment can be limited at times. But one thing that I didn’t like was the fact if I am to put my Python knowledge to use, I would have to venture outside of the learning environment and Codecademy didn’t used to provide information how.

The Learn Python 3 course made effort to help students work outside of the Codecademy environment with “Off-Platform Project”. These came in the form of Jupyter notebooks that I could download, and a page with some instructions on how to use them: a link to Codecademy’s command line course, a link to instructions for installing Python on my own computer, and a link on installing Jupyter notebooks. It’s a bit scattered.

What I didn’t know at the time was that Codecademy had already assembled an entire course covering these points. Getting Started Off Platform for Data Science is an orientation for everyone as we eventually venture off Codecademy’s learning platform. It starts with an introduction to the command line, then Python development tools like Jupyter Notebooks and other IDEs, wrapping up with an introduction to Github. This is great! Why didn’t they put more emphasis on this earlier? I think it would have been super helpful to beginners.

Though admittedly, I didn’t follow those installation instructions anyway. Python isn’t very good about library version management and the community has sidestepped the issue by using virtual environments to keep Python libraries separated in different per-project worlds. I’ve used venv and Anaconda to do this, and recently I’ve also started playing with Docker containers. For my own trip through Codecademy’s off-platform projects using Jupyter notebooks, I ran Jupyter Lab using their jupyter/datascience-notebook Docker image. That turned out to be sheer overkill and I probably could have just used the much lighter-weight jupyter/base-notebook image.

In hindsight I think it would have been useful for me to review Getting Started Off Platform for Data Science before I started reorienting myself with Python. I wouldn’t have followed it by the letter, but it had information that would have been useful beforehand. But as fate had it, it became the final course I took in the beginner-friendly section before I started trying intermediate courses.

Codecademy Beginner Friendly Python Fields

Once Codecademy got me reoriented with the Python programming language, I looked at some of their other beginner-friendly courses under the Python umbrella. I wanted to get some practice using Python, but I didn’t want to go through exercises for the sake of exercises. I wanted to make some effort at keeping things focused on my ultimate goal of learning about modern advances in machine learning.

  1. Learn Data Analysis with Pandas was my first choice, because I recognized “Pandas” as the name of a popular Python library for preparing data for machine learning. Making it relevant to the direction I am aiming for. The course title has “Data Analysis” and not “Machine Learning” but that was fine because it was only an introduction to the library. Not enough to get into field-specific knowledge, but more than enough to teach me Pandas vocabulary so I could navigate Pandas references and find my own answers in the future.
  2. How to Clean Data with Python followed up with more examples of Pandas in action. Again the course is nominally focused for data analytics but all the same concepts apply to cleaning data before feeding into machine learning algorithms.
  3. Exploratory Data Analysis in Python is a longer course with more ways to apply Pandas, including a machine learning specific section. Relative to other courses, this one is heavy on reading and light on hands-on practice, a consequence of the more general nature of the topic. And finally, this course let me dip my toes in another popular Python library I wanted to learn: NumPy.
  4. Learn Statistics with Python was how I dove into NumPy waters. After barely skating by some statics and number crunching in the previous course, I wanted a refresher in basic statistics. Alongside the refresher I also learn how to calculate common statistics using the NumPy library. And after the statistics calculations are done, we want to visualize them! Enter yet another popular Python library: matplotlib.
  5. Probability is the natural course to follow a refresher in basic statistics. They cover only the most basic and common applications of statistics and probability for data analysis, we’re on our own to explore in further depth outside of the class. I anticipate probability to play a role in machine learning, as some answers are going to be vague with room for interpretation. I foresee a poor (or misleadingly confident) grasp of probability will lead me astray.
  6. Differential Calculus was a course I poked my head into. I remembered it was quite a complex subject in school and was surprised Codecademy claimed anyone could learn it in two hours. It turns out the course should be more accurately titled “an introduction to numpy.gradient()“. Which… yes, it is a numerical application of differential calculus but it is definitely not the entirety of differential calculus. I guess it follows the trend of these courses: overly simplfied titles that skim the basics of a few things. Teach just enough for us to learn more on our own later.
  7. Linear Algebra starts to get into Python code that has direct relevance to machine learning. I know linear regression is a starting point and I knew I needed an introduction to linear algebra before I could grasp how linear regression algorithms work.
  8. Learn How to Get Started with Natural Language Processing was a disappointment to me, but it was not the fault of the course. It’s just that the machine learning systems in this field aren’t usually reinforcement learning systems. Which was the subfield of machine learning that most interested me. At least the course was short, and taught me enough so I know to skip other Codecademy natural language courses for myself.

The final Codecademy “Beginner friendly” Python course I took was titled “Getting Started Off Platform for Data Science.” I don’t think Codecademy put enough emphasis on this one.

Getting Reacquainted with Python via Codecademy

A few years ago I started learning Python and applied that knowledge to write control software for SGVHAK rover. I haven’t done very much with Python since, and my skills have become rusty. Since Python is very popular in modern machine learning research, a field that I am interested in exploring, I knew I had to get back to studying Python eventually.

I remember that I enjoyed learning Python from Codecademy, so I returned to see what courses had been added since my visit years ago. The Codecademy Python catalog has definitely grown, and I was not surprised to see most of it are only accessible to the paid Pro tier. If I want to make a serious run at this, I’ll have to pay up. Fortunately, like a lot of digital content on the internet, it’s not terribly difficult to find discounts for Codecademy Pro. Armed with one of these discount venues, I upgraded to the Pro tier and got to work. Here are some notes on a few introductory courses:

  • Learn Python 2 was where I started before, because SGVHAK rover used RoboClaw motor controllers and their Python library at the time was not compatible with Python 3. I couldn’t finish the course earlier because it was a mix of free and Pro content, and I wasn’t a Codecademy Pro subscriber at the time. I’m not terribly interested in finishing this course now. Python 2 was officially history as of January 1st, 2020. The only reason I might revisit this course is if I tackle the challenge of working in an old Python 2 codebase.
  • Right now I’m more interested in the future, so for my refresher course I started with Learn Python 3. This course has no prerequisites and starts at the very beginning with printing Hello World to the console and building up from there. I found the progression reasonable with one glaring exception: At the end of the course there were some coding challenges, and the one regarding Python classes required students to create base classes and derived classes. Problem: class inheritance was never covered in course instructions! I don’t think they expected students to learn this on their own. It feels like an instruction chapter got moved to the intermediate course, but its corresponding exercise was left in place. Other than that, the class was pretty good.
  • Inheritance and other related concepts weren’t covered until the “Object-Oriented Programming” section of Learn Intermediate Python 3, which didn’t have as smooth or logical of a progression. It felt more like a grab-bag of intermediate concepts that they decided to cut out of the beginner course. This class was not terrible, but it did diminish the advantage of learning through Codecademy versus reading bits and pieces on my own. Still, I learned a lot of useful bits about Python that I hadn’t known before. I’m glad I spent time here.

With some Python basics down — some I knew from before and some that were new to me — I poked around other beginner-friendly Codecademy Python courses.

Fusion 360 Foundational Concepts Tutorial

foundational-concepts-iconI went back into Autodesk’s Fusion 360 learning resources for a refresher and to set myself up to learn the Fusion 360 CAM modules. The last time I went through the tutorials, I had skipped the CAM functionality because I had no machine tools and were not likely to get time on anybody else’s machinery. Now that I might be able to access Tux-Lab fabrication machinery, I wanted to make sure I won’t break the machine from doing anything stupid in Fusion 360.

Before I got to CAM, though, the “Foundational Concepts” section caught my attention. I either didn’t see it the last time or it made no impression on me at the time. I went through the set of short videos and they were surprisingly informative. Most tutorials for Fusion 360 (and most other software packages in general) are happy to tell users how to accomplish their tasks. This is a slightly different twist – the foundation concepts talk about why Fusion 360 is the way it is. About how they tried to restructure a CAD package for the cloud-based future, about how they restructured the workflow to take advantage of today’s level of computation power at our fingertips, so on and so forth.

I come from a software engineering background and I’m all too aware of the fact that the end user typically has no idea what the software developer had intended as they built the piece of software. It can be argued that the end user doesn’t need to know anything about the intent if the software is sufficiently well-designed. But for something complex like a CAD package, I believe there is value in learning the motivation behind the design.

And even if the user doesn’t need to know, sometimes the user is curious and wants to know. I appreciate the Fusion 360 user education team for putting this information out there available for those who want to know.

Udemy: Product Design in Autodesk Fusion 360

udemylogoTalking with some people at the Hackaday meet taught me that there are some real fans of Autodesk Fusion 360. It was already on my to-do list to evaluate the software, but the enthusiasm pushed it to the foreground.

The first frustration is learning the software. All the resources I found are in the form of tutorial videos. I personally prefer information in written text format, but I’ll watch the videos if I have no alternatives. The videos on Autodesk’s own site is built using a Flash-based video system. I avoid Flash when I can, so as an alternative I chose the udemy.com course titled Product Design in Autodesk Fusion 360 from idea to prototype.

The quality of the course was uneven: different sections in the course were recorded by different instructors each with their own style. Fortunately, the introductory and starting sections about content creation were pretty good, teaching me Fusion 360 basics and then how to create a design. (Sections 1 through 3)

Where these sections fell short was teaching me how to fix problems. That is, beyond hitting Control+Z immediately after a mistake. I know I’ll change my mind on certain design decisions and need to adjust the file later. The class didn’t spend much time over the editing tools and I felt this gap would prove to be a hindrance to my productivity.

Section 4: Rendering, animation, and drawings were very perfunctory and barely enough to get me started. If I actually want to make use of these features, I’ll need to find additional education before I can be proficient.

Section 5: Computer Aided manufacturing(CAM) assumed the student is already familiar with machining and wanted an overview on how to use their existing knowledge inside the Fusion 360 UI. If the student didn’t already know CAM, the section would make no sense.

I think I picked up enough basics for me to start poking around on my own to fill in gaps in my knowledge, so it worked well enough as a brief overview. I’m just a bit disappointed because my expectations were higher.

“Learn Enough Command Line to Be Dangerous” notes

leclThe Ruby on Rails Tutorial was a great way for developers like me to get up and running with no prior Rails experience. However, it does require the reader to have some level of computer software skill so complete beginners could still get lost.

Acknowledging this, the author Michael Hartl has taken on the challenge of getting people up and running in the world of software development under the umbrella of his venture “Learn Enough Society” with a series of titles in the pattern of Learn Enough [X] to Be Dangerous.

I was curious about how his approach, so even though I’m not the target audience I spent some time looking it over.

His idea of starting from Ground Zero is the command line, which I agreed was a fair place to start. The reader is taken through the basics of navigating files and directories and is introduced to some basic command line utilities such as grep.

I applaud the focus on repeatedly prodding the reader to use the man command to learn more information. The manual pages displayed by man were written by programmers for programmers, which meant they tend to be densely packed with jargon difficult to pierce even for experienced users. But that’s where the information is, so one has to dive in to find answers. The tutorial tries to ease readers into man pages by teaching how to search for specific information so the reader is not overwhelmed trying to understand the full text. Baby steps.

Not being the target audience, I can’t judge how effective the tutorial is. I’m sure there are problems that didn’t stand out to me because I already knew the information, almost certainly because it didn’t stand out to the author either.

I did notice one example, though: The tutorial used the computer term argument without explanation. This might cause confusion: “I don’t want to argue with anybody, I just want to learn the command line!” (UPDATE: I e-mailed the author about this observation, and he has added explanations to the tutorial. Very cool.)

Rails Tutorial (Take 2)

With all the fun and excitement around 3D printing, I’ve let my Ruby on Rails education lapse. I want to dive back in, but it’s been long enough that I felt I needed a review. Also, during my time away, the Ruby on Rails team released version 5, and Michael Hartl’s Ruby on Rails Tutorial was updated accordingly.

Independent of the Rails 5 updates, it was well worth my time to go through the book again. On second run, I understood some things that didn’t make sense before. It was also good to look at the first do-nothing “hello app” and the second automated-scaffold “toy app” with a little more Rails knowledge under my belt. The book is structured so the beginner reader didn’t have to understand the mechanics of the hello or toy apps, but readers with a bit of understanding will get something out of it.

The release notes for the update mentioned that a few sections were rearranged for better pacing and structure, and added more exercises for readers to check their progress. Both are incremental improvements that I appreciated but neither were especially earth-shattering.

Action Cable, one of the big signature feature of Rails 5, was not rolled into the book. Hartl is handling that in a separate tutorial Learn Enough Action Cable to be Dangerous which I will go through at some point in the near future.

Towards the end of the book, Hartl introduced an optional advanced concept: using Amazon Web Services to store user image uploads. I skipped that section the first time through, and decided to dive into it this time.

I quickly found myself in a deep rabbit hole. Amazon Web Services has many moving parts designed for a wide range of audiences and it’s a challenge to get started without being overwhelmed.

Which is where I am now. Lots more exploration ahead!

“Ruby on Rails Tutorial” notes

RailsTutorial-cover-webTowards the end of the Getting Started with Rails guide, there was a link to the Ruby on Rails Tutorial. I had missed it the first time I went through the Getting Started guide, but when I reviewed the guide a second time (things made a lot more sense) I noticed and decided to check it out.

I’m very glad that I did!

For the most part, this book (available as paper edition, but the author had made it freely readable online on the web site) was exactly what I had hoped to find for Ruby on Rails. The book went through the process of making a Rails app three times, in exactly the progression I wanted:

First pass: A simple “hello world” that does nothing much but lets the student experience all the standard overhead around creating and deploying a Rails app. It’s barely a Rails lesson at all: it’s really a lesson for learning the surrounding infrastructure before digging into the meat of learning Rails.

Second pass: A simple “toy app” that does a lot… but the student is rushed through without understanding what’s going on behind the scene. It’s really a demonstration of what’s possible via Rails helpers & scaffolding and less about learning Rails. It reminds me of what the Codecademy Ruby on Rails “class” was like. A lot of incomprehensible fancy flash. However, unlike Codecademy which left me asking “now what?” Rails Tutorial follows up.

Third pass: Over 80% of the book is spent building the “sample app.” Starting from a site that can serve a few static pages and grows, bit by bit, into a mini clone of Twitter. It had creation and maintenance of user accounts, authentication, posting 140-character messages, following & un-following users, all the core bits we associate with Twitter. (Which, the author asserted, was also originally built using Ruby on Rails.)

The Codecademy Rails class left me confused and feeling like I’ve wasted my time. (About a day.)

The Rails Guides left me with a basic level of comprehension of what goes on in a Rails app. My time (About half a week) is well spent, but I didn’t feel like I could build anything on my own yet.

After the Ruby on Rails Tutorial (which took me over a week) I feel like I have all the basic tools I need to start playing in the Rails playground. I feel like I have an idea how to get started, how to experiment, understand what the experiment does behind the scenes, and debug whatever goes wrong while I experiment.

And with that, it’s time to practice using my new tools: Let’s build something!

Codecademy “Learn Sass” notes

SasslogoWhile learning Ruby on Rails, one of the things I put on my “look into this later” list was Sass. I knew it was related to CSS but didn’t know the details, I just noticed when the Rails generator created a controller, it created a .scss file under the stylesheets directory.

So when I received email from Codecademy notifying me that they have a new class on Sass… the “look into this later” became “let’s look into it now”.

Unlike Ruby on Rails, Sass is not a huge complicated system. It solves a fairly specific set of problems typical of CSS growing unwieldy as it grows with a project. It introduces some very nice concepts to keep CSS information organized. After banging my head on lots of walls with Ruby on Rails, it is refreshing to tackle a smaller-scope project and be able to understand what’s going on. The Codecademy format is well suited to teach a smaller scoped concept like Sass.

I was also mildly amused to learn that Sass is apparently written in Ruby. I don’t think it particularly matters what the implementation was, but it’s amusing to me to see Ruby applied in an entirely different way from Rails. The bonus is that, if I should try to debug or extend Sass myself, I wouldn’t be starting from scratch looking over its source code.

Being a fresh course, the Codecademy class had a few minor problems that still need to get ironed out. The flashcard example was supposed to flip on mouse hover… it never did anything for me. Too bad, because I think the effect would have been interesting.

I haven’t gotten far enough with Rails to think about making my web app pretty, but when I do, I know how to keep my style sheets manageable with Sass.

Codecademy PHP notes

elephpant

I just zipped through the PHP course in the Codecademy “Language Skills” section.

As is typical of beginner-friendly Codecademy language skill courses, it starts from the beginning with variables, flow control, etc. If the student is already aware of such basic concepts, this can get very tiresome.

This PHP class flew through basic topics. This is great for experienced programmers already familiar with the concepts and just wanted to see how PHP differs from the other languages. The downside is that, if a student is truly a beginner, they wouldn’t have received much help and would probably be lost.

I thought the class would get into more difficult topics, or areas specific to PHP and web development and… it didn’t. After covering the basics of declaring classes and class inheritance, the class stops. Even though the introduction mentioned that PHP web apps can contact databases and other server-side resources, this class didn’t cover any of the meat-and-potato of writing web applications with PHP.

Overall, the class is a superficial skimming of the surface for PHP. Good for somebody who just wants to see a quick view of PHP basics, and I appreciated it for that, but bad for somebody who actually wants to figure out how to do useful things with PHP.

I’ll have to look elsewhere for that.

Loopiness

Unrelated to the skimpy curriculum is a problem with the interactive learning development environment. In other Codecademy courses, the student writes the code and presses a button for it to be executed and evaluated. In this class, evaluation is constantly happening before the button is pressed. The upside is that feedback for errors are instant, no waiting until the end. The downside is that it’s easy to make it go into an infinite loop.

Example: If this was the goal:

$abc = 1;

while ($abc < 10)

Just before we type “<10” the state would be:

$abc = 1;

while ($abc)

Which is an infinite loop that sends the evaluation code spinning uselessly. So when the student actually presses the button later, the gear just spins.

Workaround: copy the newly-typed code into the clipboard, hit refresh on the browser to reset and reload the page, then paste the code in.