I am optimistic that one of the popular static site generators can help me reach my goals for an improved Sawppy documentation site. But the site generator itself is not enough, there are a few other details I’ll have to investigate. The primary one being ability to preview changes to pages earlier rather than later in the pipeline.

Today’s flow of editing markdown files can deliver immediate feedback because GitHub has a “Preview” tab on its built in Markdown editor. But once we’re no longer directly using GitHub’s simple Markdown transformation, we’ll lose that ability. Contributors should not be expected to set up a SSG environment on their computer in order to see the results of their work, and asking maintainers to review every change on their own SSG environment would not scale. (I say this using plural as if Sawppy has a big maintenance staff… it’s actually just me.)

The answer is to bring in tools from the continuous integration world, where tools exist to preview changes to a website before deploying live. Some use services like Netlify, which is not itself open source but there is a free tier available.

One example: look at the repository for Write the Docs website. Open one of the pending pull requests and click on “Show all checks”. One of them is “Read the Docs build succeeded!” and clicking “Details” will bring up a version of the site built with changes in the pull request. This is an interesting venue of investigation to learn more about.

This was the point where I ran out of steam, and the Write the Docs meeting ran out of time, but I have a big treasure trove of pointers to investigate and keep me busy for a while.

Other miscellaneous notes:

• Project Tetra is a hardware project that has a documentation publishing system that uses Docsy and Hugo. It also uses PlantUML and Graphvis to render diagrams.
• Markdown is not the only lightweight markup language, here’s a list on Wikipedia.
• It is quite clear Hugo is not the only documentation system out there, Wikipedia has many more in a list.

I’m glad I had the chance to learn about terminology and tools of industrial-strength documentation. They are great for their respective niches, but adopting any of them would require major changes and I’m not ready to take such a big leap just yet. Which brings us to static site generators. (SSG) This category of software tools see a lot of open source development, giving us many options to choose from.

Background: As input, SSGs take content in various formats. A set of rules and templates are applied to that content, generating as output a set of HTML and CSS (and maybe JavaScript) files that can be served by any web server. “Static” in this context means the web server does not have to run any code to modify the files, they are transmitted directly to users’ web browsers as-is. (As opposed to a dynamic systems like PHP.)

A large number of SSGs accept Markdown as a text content input format, so Sawppy’s existing Markdown documentation could be used with small modifications rather than complete rewrites. This also preserves the advantages of using Markdown, meeting the ease-of-use challenges A and B.

Every SSG offers customization of the rules and templates that it applies to content. At the minimum there are themes for cosmetic appearance, but plugins and extensions allow more extensive functionality. This is where I hope to create something that meets my challenge C including a lightweight BOM tool. Around this point Eric spoke up that the JPL Open Source Rover documentation system has a script that generates parts list from document content, but the generated dependency tree is not exposed to the viewer. I want to build upon that precedent and also make this kind of information available to rover builders.

To be pedantic, Sawppy documentation is already using a SSG because GitHub does not display raw Markdown files. A simple transformation to HTML has been applied for display. However, the reason I started this investigation is because the simple GitHub transformation is very limited. GitHub is aware of this, and as an upgrade, has built-in support for a SSG called Jekyll for generating GitHub Pages.

As another example, most of us had experience reading technical software documentation on some subdomain of ReadTheDocs. All of these content were generated by a SSG. MkDocs and Sphinx are two popular SSGs for ReadTheDocs, and a lot of the default functionality (automatic indexing, references, etc.) useful for technical software documentation would be useful for Sawppy as well.

But features like a lightweight BOM tool would be outside the scope of software documentation, so there were several recommendations for investigating Hugo. It is apparently the current darling for flexibly transforming content (including Markdown text) into varying different presentations. Associated with Hugo is Docsy, a Hugo theme focused on technical documentation.

Hugo and Docsy would be a bigger step up in complexity than something like Jekyll, but I’m optimistic that the benefit will justify the cost. I plan to use that as my starting point and expand my experimentation from there. But they’ll only be a part of the solution, because no matter the static generator I use I will still want a way to preview changes.

I outlined my Sawppy project and the challenges I want to tackle to the combined Write the Docs LA/SGVLUG meetup on the evening of October 8th. Sawppy’s current system of a loose set of Markdown files score highly on ease of contribution (challenge A) and ease of management (challenge B), but fall flat on querying dependencies (challenge C). What can I look into that helps improve information presentation without giving up the rest?

Fundamentally speaking, challenge C is not new, as it would be desirable in any large scale engineering project. The novel twist here is the desire to do it all with a system that is inviting for public contribution. As a general rule, documentarians for large engineering projects are professionals who have undergone training and have licenses for proprietary software tools.

DITA

Most such tools are excluded from consideration due to cost, but many of them deal with DITA, an open XML-based data model for authoring and publishing under the custody of the OASIS consortium. It is the standard answer to reassure customers wary of being locked in to proprietary file formats. And since it is an open format, there exists a DITA Open Toolkit to transform DITA data to desired output formats… HTML, PDF, even Markdown! There are learning resources at https://learningdita.com/

As a XML (and thus text) based format, DITA would be GitHub friendly for branching and merging. It is very flexible for creating any organization we want (creating a “Subject Schema” for DITA) but taking advantage of that power will take work. DITA Open Toolkit functions at a lower level relative to the other tools discussed later. A quick web search found many commercial offerings built on DITA (example: https://easydita.com/) but failed to find free open source counterpart.

So DITA is powerful, but that power comes at a cost, and I’ll have to decide if the cost/benefit analysis comes out in favor. This also applies to several other professional documentation concepts.

I want to improve the usability of Sawppy documentation. Keeping in mind some example problems I want solved I started my quest. To find a system to document and track these types of relationships, without losing too much of the advantages of my current system. This means every candidate system must meet the following challenges:

Challenge A: Easy to Contribute

When a Markdown file is hosted on GitHub, a potential contributor can click the “Edit” button to create a fork of the repository, make a quick fix, and create a pull request all via GitHub website. This presents an extremely low barrier to entry for contributors which is a feature I want to preserve. If contributors were required to install and learn some piece of documentation software, that would discourage many people from participating before we even talk about the learning curve for that software.

Challenge B: Easy to Manage

When using GitHub’s web interface to edit a Markdown file, visualizing the change is as simple as clicking over to the “Preview” tab of the editor. Sadly such ease can’t be matched by any system external to GitHub, but it would be nice to have some way to let a contributor see what the end results look like. Failing that, I must have a way to visualize the final impact of changes before I merge a pull request. It is unacceptable to not see changes until after merging into the main branch.

Challenge C: Easy to Query

The desired documentation system will take metadata written by document author and build an interactive presentation. This way rover builders can find information without being constrained by the current linear narrative. Here are some examples of questions I’ve received for Sawppy, rephrased in terms of wheel axle.

• (Easy: positive query) I’m on the wheel hub assembly page, and it needs the wheel axle which I guess I forgot to build. Where do I need to go?
• (Hard: negative query) My order of 8mm shafts got delayed. What can I work on now that’s not dependent on having these shafts?
• (Both of above) How can a teacher most effectively divide up rover tasks so multiple students teams can build the rover in parallel?

Challenge D: Free

It would be nice for the system to be free as in freedom, but at the very least it must be free as in beer. Design for Sawppy is given away at no cost to rover fans everywhere, there is no profit to cover monetary expense of a commercial documentation system.

Once I laid out these challenges to the group and opened the meeting to discussion, people started offering suggestions. Some professional documentarians brought up DITA as a venue for investigation.

To illustrate problems with Sawppy’s documentation, I’ll use a single component as example: Sawppy wheel axle. There are at least three entirely separate pages relating to the wheel axle:

1. The page of parts list, telling a builder to buy 8mm metal shaft.
2. The page for 8mm shaft modification, where I describe how to cut the long shaft into shorter segments. Followed by steps to turn these segments into wheel axles. Other segments were turned into steering shafts, plus those turned into rocker-bogie suspension pivots.
3. The page for wheel hub assembly, which incorporates a single segment of the 8mm wheel axle shaft.

While these three files were all linked from the index page, there’s no obvious way to retrieve the relationship between them in the context of wheel axles. I can manually add links between them, but this is time consuming and perpetually incomplete. Even worse, as the number of relationships grew, it will quickly become a maintenance nightmare.

Thus I started my quest to find a system to document and track these types of relationships without losing (too much) of the advantages of my current system.

When I decided to release Sawppy to the world, I thought briefly about how to best organize all the information I want to convey for rover assembly. I quickly fell into a state of Analysis Paralysis and, as a path out of that state, decided that it was better to have something written down whatever the format. No matter how unorganized, is still better than keeping it all in my head.

I first tried putting it in the “Build Instructions” section of Sawppy’s Hackaday.io page, but that feature has some strange and unpredictable limitations that became annoying as the length of instructions grew. The final straw was when I noticed that images and instructions for earlier steps were disappearing as I added later steps. That made me… unhappy, so I went to something else.

The second attempt is what I have as of today: a loose collection of Markdown files on a Github repository. Edited in a code editor rather than a word processor, I struggled with typos and grammatical errors as I lacked the usual automated proofreading tools present in a word processor. Still, with a large helping of assembly pictures, it was just barely enough to help other people build their own rovers.

I was painfully aware of the fact there is a ton of obvious room for improvement. This was just the “get it written down” first stage and at some point I need to revisit the various problems still open. The most significant of which is lack of structure beyond an index page with links to all the other pages. The index suggested a relative ordering that matched my personal assembly order, but that doesn’t necessarily work for anyone else. And worse, they would be stuck if they wanted to ask some specific questions my layout is unable to answer.