Launching a Flask Web App on Startup

flaskThe first few experiments with Flask has been successful. A simple user interface can be created in HTML and accessed from local as well as remote clients over HTTP. One of the downsides bit us today: a web app isn’t as easy to launch as a native local application. When running locally, we still need to get two parts up and running: the server-side code and the web browser pointing to http://localhost.

raspberry-pi-logoThis is especially problematic when the desire is to launch the app on power-up of the Raspberry Pi 3 that will run the server side. “Run on startup” is one of those tasks that have many different approaches across different flavors of Linux. Each with their advantages and disadvantages, benefits and gotchas.

The first attempt is to write a shell script that launches the Flask web server and the Chromium web browser. This works when executed interactively at the command line logged in as the default user pi. But it didn’t work when executed as a cron @reboot job, probably something to do with the fact it would run as root.

The next effort led to Raspberry Pi documentation, which pointed to the file /etc/rc.local. We seem to have better luck here – the Flask web server does launch but Chromium web browser still did not.

Further hunting eventually found this page for setting up a Raspberry Pi running Chromium web browser as a web kiosk. Not quite the project at hand, but close enough for the answer to be applicable. This meant leaving Flask startup in /etc/rc.local and editing the file /home/pi/.config/lxsession/LXDE-pi/autostart to have it launch chromium-browser with the parameter --kiosk http://localhost:5000/

Upon power-up, this Pi will now launch Flask web server and Chromium web browser to point at that server running on the same device. It works, but it’s not clear why these hoops were necessary. It’s pretty bewildering for a beginner to try to understand the why behind all these different startup mechanisms. A few hours of reading (sometimes contradictory) documentation did not make the situation any more clear. Sadly, time constraints dictate that we can’t get to the bottom of this issue today. Perhaps later.

Exploring Flask, a Python Web Framework

flaskToday’s exploration centers around Flask, a framework for building web sites with the Python programming language. The motivation was a new project that will be discussed separately, the important detail for now is that it needed to interact with some existing Python libraries and desired a user interface accessible from multiple different (and separate) devices. It seemed to make sense to turn the central device into a web server and expose the user interface to its clients using HTML.

There had been previous explorations into writing web server software using Ruby on Rails, which is not the best fit for this project. The first issue is the language – we need to talk to a Python library and there’s no need to jump through language translation layers for a simple project. The second issue is complexity – Rails is a full MVC framework for building an app on a database and we don’t need that here, either.

If we wanted the Python equivalent to Ruby on Rails, the popular choice of the day is Django. But since we don’t, Flask is the lighter-weight framework we’ll play with today. Sometime in the future, when a project is right for Django, we’ll explore using the right tool for the job.

Flask is a thin layer that’s a short step above having to call Python’s HTTP and URL classes directly, and can be very quick to get started.  A simple Hello World type of app requires only a single Python file and a few lines of code, no database configuration and project generation necessary. The developer builds up from this simple foundation. This is great for keeping things simple and easy to work with. For the most part, features that a developer would want can be added piecemeal.

The downside is that Flask also skips out on a lot of web practices that aren’t explicitly features, but are very nice to have by default. For example, the default Ruby on Rails generator for HTML <form> includes a token to prevent cross-site request forgery. This is not explicitly a feature a developer would think of adding, and could easily get overlooked especially when the forms are generated manually. Fortunately most of the Flask add-ons to generate forms – like this one – automatically includes CSRF protection.

Some caution would be warranted if the Flask application is going to be a public web site, but for a local network project prototype it should be good enough.

Twitter API Newbie Wades Into The Pool

Twitter DeveloperTwitter trends are easy to follow by searching for the hashtag. The default view given by Twitter shows the most popular tweets with that tag, and there are options to see the most recent tweets first, or browse them by photo, etc. But what if someone wants the tweets organized another way? Fortunately Twitter knows they can’t cover everybody’s desires and offers a developer API for those with creative ideas on tweets.

Twitter’s API is a “RESTful” design. It follows many of the ideas of REpresentational State Transfer (REST) but not strictly adhering to all of it. Functionally, it is built around HTTP GET and POST requests and returns information in the text-based JavaScript Object Notation (JSON) format. The details of those HTTP requests – and the JSON that it retrieves – are documented on Twitter’s developer site.

Before the aspiring Twitter app author can get anywhere, though, one needs to register with Twitter for access to these APIs. Once an app is registered, it is assigned a set of secret access keys and tokens. This allows Twitter to monitor usage and limit/disable Twitter access by individual applications.

Once registered, the code experimentation can begin. As a RESTful API communication via JSON, any web-friendly development framework can be used. (And if it is not web friendly… why would it have anything to do with Twitter?) As a starting point, Twitter has a non-exhaustive list of libraries that cater to different programming environments.

Out of this list, Python-Twitter was chosen as the first step into this world. Python language is fun and easy to experiment with. Python-Twitter installs easily via Python’s pip package manager. And yes – it also happens to be the first entry on the list.

A few commands into Python interactive console was rewarded with immediate results, though sadly also a few issues. Text of some tweets were truncated. Perhaps this has something to do with the recent increase in Twitter’s character limit? Whatever the root cause, it is tracked as an open issue in Python-Twitter’s repository.

The first few search queries returned only 15 results, even when there were definitely more than 15 answers. This turned out to be the default limit which is easily increased up to 100 results per search query. Going any further, unfortunately, is not possible with the standard search API. As of a few months ago, the only option to do more is to subscribe to the very expensive Twitter Enterprise API. This is out of the question for a hobbyist! Fortunately people have already complained and this past November Twitter responded with a premium search API to bridge the gap. Even though it is free, access is restricted so a registered developer has to apply for access.

Even if access is obtained, premium search is a new feature accessible via a new endpoint that is not yet supported by any of the Python libraries on Twitter’s list. Searching with a unique part of the API URL (“30day”) found this Python search tweet API that might be worth a look. If it doesn’t fit the bill for whatever reason, it might be time to build a custom solution.

Fusion 360 Lines Created Via API Are Not Limited To Sketch Plane

Cube on Sketch

Today’s performance tip helps avoid creating unnecessary planes and sketches when creating objects using the Fusion 360 API. First, some background: when sketching lines in Fusion 360 must be done in the context of a sketch object. This sketch object is in turn created to be aligned with a plane. It may be one of the default planes that intersect the origin (XZ plane, XY plane, or YZ plane) or a construction plane.

When sketching with the interactive UI, the user’s sketch entities are constrained to the plane of the sketch. For example, if a sketch was created on the XY plane, then all sketch entities would have a Z coordinate of zero. This makes sense to keep the data organized and easy to understand by the human looking at a 2D computer monitor.

When creating entities with the API, though, they are apparently not constrained. The first clue came from looking at the SketchLines.addByTwoPoints API – its two parameters are Point3D objects. A little experimentation confirmed the API would gladly accept 3D coordinates that lie outside the sketch plane. The example screenshot on this page is the result of drawing an one-centimeter cube on a sketch aligned to the XY plane. Only four of the eight corners (and four of the 12 edges) lie on the plane, but the API happily accepted all the information for a 3D cube.

This is very useful when creating geometries via code. Humans need to see points on a sketch stay in a plane for them to be workable on a 2D monitor screen, otherwise things get too confusing. But code doesn’t care about appearance on a screen. So go ahead – let code create complex 3D entities on a single sketch. Skipping the unnecessary planes and sketches will generate fewer API objects. Reducing object management overhead and allowing faster performance.

Accelerate Fusion 360 API Object Creation With DirectDesignType

64Cylinders

Exploring the capabilities of the Fusion 360 API will involve experiments creating objects via script. Since it’s all code, it’s easy to repeat a process many times, a natural allure of using scripts to automate repetitive tasks. The downside is that Fusion 360 (as of time of writing) has problems resulting in slow response.

The sample code below creates an array of small cylinders one at a time. (This is not the most efficient method to create an array of cylinders but merely an illustration of the problem.) The size of the array can be adjusted by changing the value of arraySize before running the script. For a 4×4 array, the results are almost instantaneous. On a modern Core i5, slowdowns can be observed in an 8×8 array. The first few cylinders are fast, but as additional cylinders appear, each new cylinder takes longer than the last.

Fusion 360 performs a lot of bookkeeping overhead to track these operations. While the majority are outside our control, we do have control over the most expensive overhead: the timeline. It is useful when we are working with hand-created designs, but unnecessary for script – if we want to change anything, we’d update the script and run it again. In the user interface, we can click on the gear in the lower right hand side and select “Do not capture Design History”. To do the same thing in script, we can set the type of the design object to DirectDesignType. (Uncomment line in sample code below.)

This mitigates but does not solve the problem. While it is now practical to generate far larger number of objects, there is still a noticeable slowdown as the object counts go up. According to this forum post, this issue has been logged as a defect for the development team. Perhaps a future release of Fusion 360 will create the last object as quickly as it created the first.

import adsk.core, adsk.fusion, adsk.cam, traceback

arraySize = 8 # Will extrude arraySize^2 objects

def run(context):
  ui = None
  try:
    app = adsk.core.Application.get()
    ui = app.userInterface

    app.documents.add(adsk.core.DocumentTypes.FusionDesignDocumentType)

    # adsk.fusion.Design.cast(app.activeProduct).designType = adsk.fusion.DesignTypes.DirectDesignType # Uncomment to turn off tracking timeline

    rootComp = adsk.fusion.Design.cast(app.activeProduct).rootComponent
    sketch = rootComp.sketches.add(rootComp.xYConstructionPlane)
    extrudeDistance = adsk.core.ValueInput.createByReal(1)

    rangeX = range(1-arraySize, arraySize, 2)
    rangeY = range(1-arraySize, arraySize, 2)

    for nowX in rangeX:
      for nowY in rangeY:
        sketch.sketchCurves.sketchCircles.addByCenterRadius(
          adsk.core.Point3D.create(nowX, nowY, 0), 0.5)

        rootComp.features.extrudeFeatures.addSimple(
          sketch.profiles[-1],
          extrudeDistance,
          adsk.fusion.FeatureOperations.NewBodyFeatureOperation)

        adsk.doEvents()

except:
  if ui:
    ui.messageBox('Failed:\n{}'.format(traceback.format_exc()))

This code is also available on Github.

Fusion 360 Script Engine Uses Python Version 3

One of the differences between Python 2 and 3 was changing print from a statement to a function. This means a Python beginner like myself immediately runs into the breaking change trying to print “Hello World!” to the console. (print "Hello World!" versus print("Hello World!")) It sets the tone pretty clearly up front: when working with Python, developers need to know which version they’re writing for.

Which is why I was very puzzled when I searched for this information in the Autodesk documentation and came up empty-handed. The most obvious place in my mind would be the “Python Specific Issues” page of the user’s manual, but it wasn’t mentioned there. Searching various other combinations of terms – on and off Autodesk’s web site – failed to find a straightforward answer. Given the relatively young age of Fusion 360’s Python scripting support, I would expect them to use Python 3 but I wanted confirmation.

Well, if I can’t find it in documentation, there’s always looking at the code itself. And a tiny baby step beyond the simple boilerplate script generated by Fusion 360. It’s not quite printing “Hello World” but it’s almost that simple.

First I imported the Python sys module for querying system parameters.

import sys

Then I changed the boilerplate message box output string to include the version number.

ui.messageBox("We're running on Python " + str(sys.version_info.major), "Version detection")

Executing the script confirms the scripting engine is running Python 3.

F360Python3

Once the web search engines index this post, people who have this question in the future will know the answer just by reading the title of this post on the search results. They won’t even need to click on the link to read this page.

(This really simple bit of code is almost not worth committing to Github… but it’s a slow day so it was pushed up to be publicly available.)

Qt Quick with PyQt5 on Raspberry Pi

QtLogoThe prime motivation for me to go through Qt licensing documentation and installing Qt Creator IDE was to explore the new UI infrastructure introduced in Qt 5 under the umbrella of “Qt Quick“. As far as I can tell, this is an entirely different system for creating user interface of a Qt application. Built with modern ideas such as OpenGL graphics acceleration for animation effects and UI layout declared with a text-based markup language QML (probably stands for Qt Markup Language.)

Up to this point my experience with building graphics user interface in Qt was with the QWidget-based infrastructure, which has a long lineage in past editions of Qt. Qt Quick is new for Qt5 and seem to share nothing in common with QWidget other than both a part of Qt5. Now that I’ve had a bit of QWidget UI work under my belt I wanted to see what Qt Quick has to offer. And this starts with a smoke test to make sure I could run Qt Quick in the environments I care about: Python and Raspberry Pi.

Step 1: Qt Creator IDE Default Boilerplate.

Once the Qt Creator IDE was up and running, I followed the Qt Quick tutorial to create a bare bones boilerplate Qt Quick application. Even without any changes to the startup boilerplate, it reported error messages complaining of missing modules. Reading the error message, I looked at the output of apt list qml-module-qtquick* and installed the ones that sound right. (From memory:qml-module-qtquick2qml-module-qtquick-controls2, qml-module-qtquick-templates2, and qml-module-qtquick-layouts)

QML CPP

Once the boilerplate successfully launched, I switched languages…

Step 2: PyQt5

The next goal is to get it up and running on Python via PyQt5. The PyQt5 documentation claimed support for QML but the example on the introductory page doesn’t quite line up with the Qt Creator boilerplate code. Looking at the Qt Creator boilerplate main.cpp for reference, I translated the application launch code into main.py. This required sudo apt install python3-pyqt5.qtquick in addition to the python3-pyqt5 I already had. (If there are additional dependencies I forgot about, look for them in the output of apt list python3-pyqt5*)

QML PyQt

Once that was done, the application launched successfully on my Ubuntu desktop machine, albeit with visual appearance very different from the C++ version. That’s good enough for now, so I pushed these changes up to Github and switched platforms…

Step 3: Raspberry Pi (Ubuntu mate)

I pulled the project git repository to my Raspberry Pi running Ubuntu Mate and tried to run the project. After installing the required packages, I got stuck. My QML’s import QtQuick 2.7 failed with error module "QtQuick" version 2.7 is not installed The obvious implication is that the version of QtQuick in qml-module-qtquick2 was too old, but I couldn’t figure out how to verify version number is indeed the problem or if it’s a configuration issue elsewhere in the system.

Searching on the web, I found somebody on stackoverflow.com stuck in the same place. As of this writing, no solution had been posted. I wish I was good enough to figure out what’s going on and contribute intelligently to the discussion!

I don’t have a full grasp of what goes on in the world of repositories ran by various Debian-based distributions, but I could see URLs flying by on-screen and I remembered that Ubuntu Mate pulled from different repositories than Raspbian. I switched to Raspbian to give that a shot…

Step 4: Raspberry Pi (Raspbian Stretch)

After repeating the process on the latest Raspbian, the Qt Quick QML test application launches. Hooray! Whether it was some configuration issue or out of date binaries we don’t know yet for sure, but it does run.

That’s the good news. Now the bad news: it launches with the error:

JIT is disabled for QML. Property bindings and animations will be very slow. Visit https://wiki.qt.io/V4 to learn about possible solutions for your platform.

And indeed, the transition between “First” and “Second” tabs were slow. Looking on the page that it pointed to, it looks like the V4 JavaScript engine used by Qt for QML applications does not have JIT compilation for Raspberry Pi’s ARM chip. That’s a shame.

For now, this excludes Qt Quick as a candidate for writing modern responsive user interfaces for Raspberry Pi applications. If I want to stick with Qt and Python, I’m better off writing Qt interfaces in the old school QWidget style. We’ll keep an eye on this – maybe they’ll add JIT support for Raspberry Pi in the future.


(The source code related to this blog post are publicly available on Github.)

First Use of Python Threads is Quickly Followed By First Crash… in Qt

QtLogoBased on experience, I fully expected my first venture into Python multi-threading to run into problems. I just didn’t know exactly what that problem would look like. I had hoped that my first Python threading bug would be friendlier to understand and debug, just as the Python programming language has been friendlier for a beginner to write.

Sadly, no such luck. After I started using my GPIO class, with the debounce routine powered by threading.Timer, the crash is a very unfriendly Segmentation Fault. Surprisingly, it’s not the full “Segmentation Fault (core dumped)” so there was no core for me to debug with. No matter, it’s my program and I can relaunch it under gdb and try to dissect each instance of the crash.

The symptoms are consistent with a multi-thread timing issue:

  • It is unpredictable. I could use my program for several minutes without a problem, or the program might die within a few seconds of launch.
  • When something does go wrong, the actual point of failure that crashed in gdb is deep in the bowels of the system far from the actual problem.
    • In one case, a Python thread worker without any of my code on the stack.
    • In another case, internal system corruption error “Error in `python3': corrupted size vs. prev_size
  • The state of the rest of the system is also unpredictable. Using gdb’s “info threads” command I could get an overlook of the rest of the program, and it looks different every time.

I thought my timer-related code was simple: it checks the state of the input pin, and if the pin level has changed (after the debounce wait) also change the appearance of the visual indicator. The visual indication change was done by changing the stylesheet on the QWidget on screen. One of the error messages gave me my first clue that was my bug.

Could not parse stylesheet of object 0x21f6f58

Going back to look at the state of the other threads with this in mind, I saw they all had something in common: the main UI thread at the point of the crash is working on some part of visual rendering.

So we have a candidate explanation: I had been updating the QWidget stylesheet in the timer callback, which executes on a thread that is not the main Qt UI thread. So when this occurs when the UI thread is in the middle or rendering the QWidget, it would see the data structure change out from under it by the timer thread. This is not good and could explain the crash.

To test the hypothesis, I modified my program to perform stylesheet updates from the timer callback thread and do it frequently. Tens of times per second versus once every few seconds in my normal execution. The program now crashes consistently within a few seconds of launch which I saw as good confidence builder for my hypothesis.

The next thing we need is a fix. I need to make sure the stylesheet update occurs on the UI thread, and it seems simplest to post a message to the UI thread event queue. I can accomplish this by emitting a signal from the timer thread and have that go to a slot on the UI thread via a Queued Connection. The slot executes on the UI thread and can safely update the stylesheet there.

After this modification, the test program is now only emitting a signal from the timer callback thread to queue a stylesheet update, instead of performing the actual update itself on the timer thread. And now the test program no longer crashes.

I started debugging this believing I ran into a problem deep in the bowels of Python implementation internals. It turns out my clumsy use of threads had confused Qt.

Lesson learned, moving on to the next lesson…

(The project discussed in this blog post is publicly available on GitHub. The threading fix is in this commit.)

 

Learning Timers: Qt QTimer and Python threading.Timer

QtLogoWhen I interfaced my PyQt application to the Raspberry Pi GPIO pins, I ran into a classic problem: the need to perform input debouncing. The classic solution is to have the software wait a bit before deciding whether the input change is noise to be ignored or not. A simple concept, but “wait a bit” can get complicated in the world of GUI programming. When writing simple programs, we can probably get away with a literal wait by “going to sleep” for a little bit. But we don’t have that luxury in the world of GUI programming – going to sleep would freeze everything in the program. In general, users do not appreciate their UI becoming frozen and unresponsive.

Python LogoThe solution: a timer. In a Windows application, the programmer can use the operating system timer and do their “after waiting a bit” tasks in response to the WM_TIMER message. I went looking for the Qt equivalent and found several timer-related features. QTimer, QBasicTimer, and QObject::startTimer(). Thankfully, the documentation also provided an overview describing how they differ. For debounce purposes, the most fitting mechanism is a single-shot timer.

Unfortunately, when I tried to use it, I received an error message telling me I could only use Qt timer objects from code launched with QThread. Which apparently wasn’t the case with code running under the context of a QWidget within a QApplication.

I had hoped the Qt timers, working off of the QApplication event queue, would stay on the UI thread. But it appears they have to have their own QThread. I could put in more time to figure out how to get Qt timers to work, but I decided to turn to Python library instead. If I have to deal with multi-threading issues anyway, there’s no reason to avoid Python’s Timer object in the threading library.

It does mean I had to review my code to make sure it would be correct even if called from multiple threads. Since the important state are the status of the GPIO pins, they are handled by the pigpio library and the code in my app should be fairly safe. I do set a flag to avoid creating multiple Timer objects in the case of input bounce. If I cared about making sure I only ever create a single Timer, this flag should be protected against cross-thread access. But reviewing the code I decided it was OK if a few Timer ends up being created – the final result of reading the GPIO pin should still be the same even in the case of multiple Timers doing duplicate work.

(The project discussed in this blog post is publicly available on Github.)

Notes on “ZetCode’s PyQt5 Tutorial” From a Windows Developer.

QtLogoOnce I decided to learn how to create a GUI application using the PyQt5 Python binding for the Qt framework, I looked online for some resources to get started. The reference guides for PyQt5 and Qt version 5 itself seems to be fairly robust, but I needed a little push to get over the initial learning curve to understand where to look for what I need.

The Python Foundation’s wiki for PyQt tutorials has a fairly long list. But when I looked, only one explicitly states it is for PyQt5. So off I go to Zetcode’s PyQt5 Tutorial. It was a very bare-bones tutorial that might be a bit too bare for a complete beginner trying to learn GUI programming. But for somebody who already knows the general concepts of a windowing graphics interface system, it is a quick primer to learn the vocabulary.Python Logo

Since I’ve worked with various flavors of Windows frameworks (from raw Win32, to MFC, to WPF) I just needed a tutorial to help me connect concepts in my head to the terminology used in Qt. For example: in Qt, when something happens, a notification called a “signal” occurs. A signal can be connected to a “slot” which can then respond to whatever just happened. Once I learned this, I was able to translate the concepts into my head: a “signal” is an event that could be raised, and a “slot” is an event handler. Once I got this and a few other basic “Rosetta Stone” translations in my head I could switch to the reference documentation to find answers.

One important note: Even though it is labelled as a PyQt5 tutorial, Zetcode’s tutorial is actually pretty much the PyQt4 tutorial updated with the changes to syntax needed for PyQt5. It doesn’t actually cover anything that is new in Qt5. For me this is fine, because the “old way” covered in the tutorial is probably what I’ll end up using when I go even further back in time for the ROS flavor of Qt.

But know there’s no coverage of the Qt5 advancements in declarative interface construction, hardware-accelerated rendering, etc. Anybody who wants to learn the new toys of Qt 5 would have to look elsewhere.

Qt + Python = GUI for Raspberry Pi Project

Since the mission of the Raspberry Pi foundation is to “put the power of digital making into the hands of people all over the world” there is no shortage of options for programming the Pi. We have at our disposal many choices in programming languages, each with multiple application frameworks, and a large community of Raspberry Pi users for support.

QtLogoFeeling overwhelmed with options, I chose the one that best lines up with my long-term goal of getting up and running on ROS. The ROS plug-in architecture for operator GUI is rqt, based on Qt. And like much of ROS, the user has the option of working with rqt in either C++ or Python. Since I had started dabbling with ROS in Python before getting distracted, I thought the combination of Qt and Python would be a good direction to go.Python Logo

The Qt framework itself is aimed at C++ developers, and its documentation is written accordingly. Fortunately there are translation layers (language bindings) for Python. The one that seems to be the most mature is PyQt with a long list of resources, books, and online tutorials.

The next decision to make is which version to start learning. Browsing through the resources, it looks like Qt 4 is the mainstream version and Qt 5 is the new shiny. Since ROS is still in the midst of transitioning from Python 2 to Python 3, I assume rqt would be relatively old school as well. No matter which one I choose, there’ll be differences I have to tackle whenever I get around to diving deep into ROS. On the assumption that the latest and greatest versions are also the most polished (an assumption based on how Python 3 cleaned up a lot of architectural messiness of Python 2) I thought I’d start learning with the latest releases and make adjustments later as I need to.

So: Qt 5 and Python 3 it is, with the help of PyQt5 binding. Which is easily installed on a Raspbian Stretch system by installing the packages “pyqt5-dev” and “pyqt5-dev-tools”.

Less Grumpy About Python 3’s Break From Python 2.

Python LogoI spent more than a decade and a half in a software engineering job where backwards compatibility is a must, all day, every day. It’s very disorienting for me to switch to a world where such a sacred commandment would be discarded as when Python 3 was launched with severe incompatibilities from Python 2.

As a novice to Python, I was not aware of the problems that Python developers tried to leave behind in the break and I’m also ignorant of the positives Python 3 offered as a benefit to balance the cost. All I saw was the aftermath of an incredibly disruptive change that, almost a decade after it started, is still far from complete.

Part of my motivation to learn Python is ROS, which is still on Python 2. This meant I’ll have to deal with the transition costs even though I’m new to the scene. This was not a happy introduction to the world.

Fortunately the lengthy transition also meant there’s been time for people to voice similar concerns and for the people behind Python to write up their explanation. There’s a fairly lengthy explanation on Python.org titled “Should I use Python 2 or Python 3 for my development activity?” which links to an even lengthier “Python 3 Q&A” that has evolved over the past five years.

After reading it all, I see their point. Especially the parts around Unicode support. I can see how changing behavior of a fundamental concept like strings in a language can’t be done without breaking compatibility, and I can understand why it is a required transition for the language to be successful in the rest of the world beyond the English-speaking communities.

I don’t have to be happy about it, but I can at least understand it, and that does help to make me a little less grumpy when I have to deal with the aftermath.

Scratching the Surface of Python Libraries

Python Logo
Python logo from the Python Foundation

It’s somewhat embarrassing to admit my first impressions of Python came from programmer humor. The first one I can recall is xkcd #353. Another one that made an impression was the “If programming languages were essays” series.

The common thread is the tremendous library of tools available to python programmers just a “import” command away. I think the hard part of me getting up to speed in Python won’t be the syntax or the constructs, it’ll be getting to know what libraries are available. I foresee my biggest hurdle to Python productivity would be spending three hours writing something before it occurs to me to look for a library where someone has already done the work, find it, and finish the task in 15 seconds.

It makes sense that this ecosystem of libraries is a huge strength to Python and also an equally huge liability and a hindrance in transition to Python 3. By breaking backwards compatibility, Python 3 had to rebuild this ecosystem of libraries from scratch. It’s hard for people to find motivation to use the new Python 3 – with its limited set of ported libraries – when they could stay with the comfortable group of friends they already know in the Python 2 world.

To learn the ropes, there’s nothing like diving right in. I wanted to write a simple script to handle a task I had: enumerate video files in my collection and call FFmpeg to convert them from their old formats to the modern MPEG 4 Video format (h264 + AAC). Most of what I need to do for walking the directory tree comes from “import os” and “import os.path“. Calling FFmpeg turns out to be trivial after “import subprocess“. And the final icing on the cake: a library to parse command-line parameters using “import argparse“. Argparse even has a HOWTO tutorial to walk me through using it.

In less than an hour, this complete novice was able to build a decently robust Python script for video conversion.  Color me impressed.

tumblr_lltzgnhi5f1qzib3wo1_400

Some Python Points of Interest to a C/C++/C# Programmer

Python Logo
Python logo from the Python Foundation

Zipping through the Python 3 tutorial from the Python Foundation, some things stood out more than others. Coming from a background of work in C, C++, and C#, I found the following items interesting:

Strings

String handling is something every programming language has to deal with and a good way to feel how the language designers solve problems. C approaches bare metal and has historically proven to be error prone. C++ offers the choice of going with C or taking the performance overhead of string libraries in exchange for reduced headaches. C# reflects a more recent school of thinking where the programmer is prevented from shooting themselves and all their customers in their collective feet. Python is in the last bucket and I’m happy to see it.

A different ‘for’ loop

Python’s for loops always iterate over items of a sequence, and not based on an integer index like I’m used to from C and C++. It feels like foreach from C# but hopefully without the negative performance impact.

An ‘else’ block after a loop?

The ‘else’ block of a loop was a novel idea to me. I’ve never had this tool in my toolbox and my brain isn’t used to structuring code to take advantage of this mechanism, but I am optimistic I will find it useful.

Python classes

The Python designers took a very different path to implement classes than any other object-oriented language I’ve learned. The tutorial began with some background, a big wall of text that I originally thought I’d skip thinking it’d be repeating things I already knew. Now I’m glad I put in the effort to read it. Python classes look familiar at first glance, but after reading about the nuts and bolts, I know I’ll get into trouble if I had treated them the same way as C++ classes.

I expect I’ll still get into trouble, but now I’m optimistic I will understand why and be able to dig my way out.

Learning Python 3 (not 2) from Tutorial by Python Foundation

Python Logo
Python logo from the Python Foundation

I took the Codecademy Python class several months ago when I surveyed various technologies. It appears I didn’t write a short note about it which is too bad because I now want to look over what I thought at the time. In my vague memory, they wrote it for the absolute beginner and spend a lot of time teaching basic computer programming concepts. (Variables, flow control, etc.)

Python is again as a topic of interest for me. It is one of the primary languages for working with ROS and also in a lot of machine learning/AI fields. (OpenAI gym is Python, Google’s TensorFlow is primarily focused Python, etc.) I think I should try to gain proficiency in the language but I didn’t want to go through a beginner-focused class again.

To review Python from a different starting point, I started going through the Python Foundation’s own tutorial for Python 3.6. I quickly determined the target audience are people already familiar with programming in another language and want to get up to speed in Python, which perfect for me! It was quite useful for me when the authors described Python features in terms of how it is similar to or different from C++ features.

On the flip side, I was reminded why I was down on Python: the incompatible break between Python 2 and Python 3. They’re both fundamentally the same language but in real world usage they are now effectively two different languages. Python libraries written for Python 2 could not run unmodified on Python 3, and vice versa. So they will have different install instructions, etc. Over eight years after the initial release of Python 3 the entire ecosystem is still undergoing the pain of  this transition.

OpenAI and TensorFlow appears to support both Python 2 and 3, with separate setup and usage instructions. ROS has the official  ambition to move to Python 3 but is currently still stuck on Python 2 due to the existing base of Python libraries.

Despite the temptation to stick with Python 2 for ROS, I decided to jump into the Python 3 tutorial and I’ll deal with Python 2 weirdness as they come up later.