As I’m reading through several Python Enhancement Proposal (PEP) documents, I have to keep in mind that I don’t have to understand everything on my first read through. And I also have to keep a goal in mind so I don’t get too distracted. For this session, the motivation was copy/pasting “async with” from example code without having any idea what those keywords meant. PEP 380 was the last in my list of self-assigned prerequisite readings, but now it’s time for the main attraction: PEP 492 Coroutines with async and await syntax. After completing this study session, I have a much better understanding of what “async with” does.

It was interesting to follow Python evolution on this front. My prerequisite PEP reading all discussed generators and coroutines as a specific type of generator. In PEP 492 they laid out the reasons for separating out coroutines as its own concept. The underlying implementation of coroutines still uses many pieces of generator infrastructure, but the language will now treat them as different things. As part of this evolution, a large part of this document discussed is devoted to “Design Considerations” of alternative approaches, and “Transition Plan” for maintaining compatibility with pre-492 coroutine syntax.

One decision I found disappointing was that debugging features are disabled by default. I understand the motivation to ensure debugging features do not impact production code, but I think leaving them out completely is going too far. Beginners who most need feedback from Python runtime are not going to know they need to set an OS environment variable PYTHONASYNCIODEBUG.

But I am in full support of another decision: there’s no comprehension shorthand:

Syntax for asynchronous comprehensions could be provided, but this construct is outside of the scope of this PEP.

I don’t like Python comprehension shorthands and I’m glad there to see this PEP did not add one. It’s possible this text meant only exactly what it says, but in my professional career I’ve used “out of scope” as a polite rephrase of “I don’t like this idea and I’m not doing it.” It made me smile to think the PEP author might be doing the same here.

Reading through PEP 492 concludes this particular study session, and the knowledge informed updated goals for my MX340 CircuitPython project.

I’ve dipped my toes into writing Python code for asynchronous execution, and encountered a lot of new concepts that I felt I need to study up on. One of the keywords I wanted to understand better was “with“, which took me to context managers. Another item on the list of mystery was “yield from“. I recently learned “yield” in the context of coroutines, and I knew “from” in the context of loading Python modules, but they didn’t make sense together. Thankfully I found PEP 380 Syntax for Delegating to a Subgenerator, where my confusion was explained by the fact it had nothing to do with loading modules.

When I read up on coroutines, it took effort for my brain to absorb the concept of code execution interleaved between caller and callee. It was a foreign enough concept my brain didn’t flag a consequence of Python’s special treatment: the “yield” relationship is limited to one layer of interaction. What if we wanted to refactor a chunk of code, that included a “yield“, into another coroutine? Things quickly get messy. If only there’s a way to extend yield to multiple levels, and this is the problem PEP 380 wants to solve with “yield from“:

The rationale behind most of the semantics presented above stems from the desire to be able to refactor generator code.

Reading through the PEP, I was not happy to see “StopIteration” exception was used to convey a return value out of “yield from“. I was taught in the school of “exceptions should be exceptional” and here it is just a normal non-exceptional code return path. My initial reaction was tempered by learning StopIteration is how Python iterators (which are used all the time) signal a halt. I think my instinctive negativity came from experience with languages where exception handling incurs a significant performance overhead. Judging from what I’ve learned here, either Python exceptions incur no significant penalty or Python designers feel it is an acceptable cost.

For what it’s worth, I was not alone in my negative impression. Using StopIteration to convey return value was also disclosed under “Criticisms” sections of PEP 380 and dismissed as “without any concrete justification”. Shrug. But I was thankful another criticism was dismissed: looks like there was a suggestion to use syntax of “yield *” and I’m glad it didn’t go in that direction because it’d end up as another special syntax very difficult to look up. Searching on * would be a disaster as it is popularly used as a query wildcard character. From this perspective “yield from” is far superior and it only meant typing three more characters. I’m much happier with this approach and I’m glad to see it as I continue my PEP study session with PEP 492.

I don’t like Python shorthands that are so short, it leaves a beginner up a creek without any keywords they could use for searching and learning. But I’m OK with shorthands that clean up code while still leaving something for a beginner to look up. Such is the case with PEP 343 The “with” Statement. I’ve been using “with” in Python for a while, but never really sat down to understand what’s going on when I do. Thankfully there is a keyword I could use to find appropriate documentation.

My introduction was in the context of example #2:

with opened("/etc/passwd") as f:
    for line in f:
        print line.rstrip()

Opening a file for data operations, “with” guarantees all cleanup will also be handled behind the scenes. PEP 343 explains the problems it intended to solve, and explaining how this convenience is handled behind the scenes. There were two explanations that I could follow. The first explains using an implementation specified with a set of methods with special names “__enter__()” and “__exit__()“. I understood Python will look for these names under conditions specified in PEP 343. Then the same concept was rewritten in a way that built upon PEP 342: a context manager called upon via with can be implemented as a generator that calls “yield” at a single point within a “try/finally” block. This neatly packages all associated components together. Any setup code runs before “yield“. Within the “try” block, “yield” hands control over to client code. (In the above example, the “for” loop reading text in a file line by line.) Then code inside either “except:” or “finally:” can cleanup after client code completes.

I like this pattern, ensuring setup and cleanup code can be kept together while allowing other code to run in between them. While I have not yet fully absorbed Python generators, I think I understand enough of this particular application to appreciate it.

Coverage of this topic in the official Python tutorial is under “Predefined Clean-up Actions” within the “Errors and Exceptions” section. As appropriate for a tutorial, it focuses on how to use it and how it’s useful and leaves all the history and design thinking and implementation nuts and bolts to PEP 343.

Next lesson: what did it mean when I saw “yield from” instead of just “yield“?

When learning about a Python feature like coroutines and generators, I found it instructive to flip back and forth between different ways a feature is represented. It’s nice to get the context of a feature and its evolution by reading its associated Python Enhancement Proposal, and it’s good to see how the official Python tutorial presents the same concept after the PEP process was all said and done. However, I want to take a side detour because the generator tutorial section was immediately followed by generator expressions and that made me grumpy.

Some simple generators can be coded succinctly as expressions using a syntax similar to list comprehensions but with parentheses instead of square brackets.

I am personally against such alternate syntax for the reason they are extremely hostile to people who don’t already know it. When I came across a generator and didn’t know what it was, I was able to search for keywords “yield” and “send” and get pointed in the right direction. But if someone comes across a generator expression or a list comprehension and didn’t know what it was, they have nothing to search on. The expression is enclosed by parentheses or square brackets, commonly used throughout the language. Inside the expression are normal Python syntax, and searching on “for” would just get to those features and not anywhere close to an explanation for generator expression or list comprehension.

I get that whoever pushed this through Python loved the option to “code succinctly” but my counter position is: No! Go type a few more characters. It won’t kill you, but it’ll be tremendously helpful to anyone reading your code later.

Here’s an excerpt from the list comprehension section of Python tutorial:

[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

As a Python beginner, I had no idea what this syntax meant when I first saw it. I see two “for” loops, but this isn’t just a “for” loop, I see an “if” statement, but I didn’t know what that decision affected. (There’s no obvious “then” in this if/then.) Beyond those keywords, there are just variables and parentheses and square brackets. I had nothing to put into a search to point me towards “list comprehension”.

Python list comprehension tutorial said the above was equivalent to this:

combs = []
for x in [1,2,3]:
    for y in [3,1,4]:
        if x != y:
            combs.append((x, y))

If I didn’t understand this code, I could search “for” and learn that. Same with the “if“, and I can see the result of the decision affected whether the value was appended to a list. This is way more readable, it wasn’t even that much more typing.

Was list comprehension (and generator expression) worth adding to Python? I guess enough people in decision-making positions thought yes, but I disagree. I don’t understand why Python is considered beginner-friendly when horribly beginner-hostile things like this exist. I don’t think they should be in the language at all, but that ship has long since sailed. I can only shake my fist, yell at cloud, then return to my study.

Related: I made a similar rant on impossible to search JavaScript short hands

I think I’ve worked my sparkly distraction out of my system, time to return to my Python study. This was motivated my CircuitPython experiments running on RP2040 microcontrollers. CircuitPython may be a reduced subset of Python, but it nevertheless incorporated many concepts that I have yet to grasp. Thus the study session, which dug through multiple PEP (Python Enhancement Proposal) design documents. Here are my notes after reading PEP 342 Coroutines via Enhanced Generators.

I was not familiar with coroutines, but I found a helpful explanation within Python glossary: “Coroutines are a more generalized form of subroutines. Subroutines are entered at one point and exited at another point. Coroutines can be entered, exited, and resumed at many different points.” I was familiar with functions, which has a single entry point and multiple exit points. (Each return is an exit.) When I read about “resume” in context, my first thought was of a function calling another. The parent caller function pauses execution waiting for the child callee function to run. Which is true, but Python coroutines have more points of interaction with each other. The caller can send additional data to the callee, which receives that data via yield. And they each maintain their internal state while this went on.

Why do we even want this? At surface I thought the same could be accomplished by standard nested loops, but example #2 in the PEP (JPEG contact sheet creator) helped me understand. Yes, maybe the pattern of execution could be replicated by nested loops, but that meant a single function has to track all variables involved in every nested loop. A coroutine, in contrast, can be written to encapsulate information for just one layer.

Here’s my pseudocode to replicate example #2 with nested loops:

for each page of contact sheet
  for each thumbnail on contact sheet
    for each JPEG
      generate thumbnail
    add thumbnail to contact sheet
  write page of contact sheet

If I want to process a bunch of JPEG differently, resulting in a different summary output JPEG, I would write a new function that has the different second loop but has the same innermost and outermost loops. With coroutines, I can get the same result by swapping out the thumbnail_pager coroutine and continue using the rest without making any changes to them or duplicate code.

I think I see the advantage here for independent code modules, but it’ll take a while for my brain to adapt and add this tool to my toolbox. During this transition period I’m likely to continue writing my code as nested loops. But at least this understanding helped me understand Python context managers. But before that, a complaint from grumpy Python student.