Quick Look: Frédéric Jelmoni’s Reproduction du Rover Mars 2020

Our world tour of fellow 3D-printed fan rovers started in Sweden with Jakob Kranz’s Mars Rover project. The next stop is France, or at least, a French-speaking nation: Reproduction du Rover Mars 2020 by Frédéric Jelmoni. (Thanks to Quinn Morley for passing this one along.) This project is still working up to the first complete iteration, and Jelmoni wants to finish V1 before publishing details so we have to be satisfied with glimpses on the web site and YouTube videos.

Similar to what I did with Sawppy, Jelmoni took orthographic views of Perseverance rover into Autodesk Fusion 360 and dimensioned a layout to scale. This ensures the rover has correct proportions. Looking at the screen shot of his layout, it looks like his wheels have a diameter of 125mm. This is slightly larger than Sawppy’s 120mm but in practice our rovers should be effectively identical in size.

While the body of this rover is built out of aluminum extrusion beams like Sawppy, the suspension arms are built from metal tubes instead. Based on color, my guess is aluminum. This gives an appearance more faithful to the original. The focus on visual fidelity is a consistent theme throughout this project. My Sawppy project gave up a lot of visual fidelity in favor of 3D printing practicality, so our two rover projects reflect different priorities.

A good example are the wheels. These are absolutely gorgeous! The wheel spokes replicate the inner curvature of each spoke, skipping the outer curvature visible in the online 3D model. They look great but I worry about their strength which was why I forfeited fidelity for my own rover wheel design. The 3D printer settings needs to be very well dialed in for optimal layer adhesion, otherwise these spokes are liable to fracture along layer lines. Aussie Sawppy explored a similar idea and the spokes fractured exactly in the way I feared they might. TeamSG blamed it on cheap PLA so I’d be curious if they’ll retry with different filament. Similarly, I’ll be curious to see how this design works out for Jelmoni. Especially if there will be any results from stress testing these wheels.

I loved seeing the rocker-bogie differential design used here. I don’t think the main differential body is 3D-printed. It almost looks like MDF (painted black) but I’m not confident on that guess. Many of the linkage pieces are 3D printed, the most illuminating part being the eye bolt used as the differential bar hinge. This is a good way to handle the multi-axis loads, but I find it curious that it is only used on one side of the link and the rocker side has only a single degree of freedom hinge. Perhaps multiple degrees of freedom movement is not as necessary as I thought it might be.

I’m also curious what material is being used for this rover’s flat body sides. These flat textured panels appear to be commodity items and not 3D printed. Which is great because the power of 3D printing is largely wasted printing large flat sheets. They are best at building unique shapes tailored to an application at hand. A focus on that uniqueness to create a clean minimalist design leads to the Bricolabs rovers.

Quick Look: Jakob Krantz Mars Rover

Since the time I’ve published my Sawppy rover project, I’ve learned of a few other rover projects with no relation to Sawppy (other than being inspired by the same rovers on Mars) but also utilize 3D printing. These are far more affordable than trying to duplicate the mechanical workings of a rover in LEGO. This collection also happens to hail from around the world. The first stop of this world tour is Sweden, home of a rover published by Jakob Krantz titled simply “Mars Rover”. It uses commodity PVC pipes for suspension structural members instead of the aluminum extrusion beams I used on Sawppy. The overall size is slightly larger than Sawppy, with different proportions. For example, the wheels and body appear to be smaller than proportional to Curiosity. But it is recognizably and undeniably inspired by NASA JPL Mars Rovers.

Full rocker-bogie is presented and accounted for, including 3D-printed links for the differential. Sawppy didn’t use 3D printing for these parts, it used the same hardware used on the JPL Open Source Rover. They are adjustable suspension link turnbuckles from the remote control car world, and I have yet to think up or find a 3D printed design that worked as well as those turnbuckles. It’s not clear how Krantz’s design handles rotation across multiple degrees of freedom and browsing the published CAD file only shows gaps where some critical components of the joint (bearings? bushings?) would sit. How these differential links function remains a mystery to me for now.

The published parts list included several bearings, all of which appeared to be used in the rocker-bogie suspension. Examining its CAD file confirmed that all wheel forces are sitting directly against the drive motor gearbox, and all corner wheel forces are directed into the steering servos. I prefer to have ball bearings handling such forces instead of the motor gearboxes directly, and designed Sawppy to use a pair of 608 ball bearings at each rotational joint. Perhaps overkill, but it’s what I like to have.

On the electronics side, Krantz’s rover uses ESP32 for its brains. As a powerful and affordable microcontroller with wireless capability, it is a very good choice for robotics projects. Especially since it is now one of the supported Micro-ROS architectures for interfacing with ROS2. I haven’t seen any interest from Krantz about putting ROS2 on his rover, but the option is there if he wants to.

And there’s plenty of time for future work. Like Sawppy, this rover is a long-term project of Jakob Krantz who has been working on his rover for at least a year. Or at least, there were two Hackaday writeups for his project a year apart. It is a labor of love and I know that well. It’s a journey that Frédéric Jelmoni has recently embarked on for a younger rover project focused on Perseverance.

Window Shopping LEGO Rovers

After seeing circuit boards being used as structural members in the 4tronix M.A.R.S. rover, I realized that I had forgotten a very fertile field: LEGO! My interest in mechanical systems today is something I give full credit to the LEGO sets I had growing up. My LEGO creations grew more complex as I grew older, limited only by my toy budget. However, once I got up and running on 3D printing objects of my imaginations, I haven’t touched my big box of LEGO since. I quite enjoy the freedom of not being constrained by what LEGO pieces are available.

Among currently available LEGO sets, there is a tiny little Mars rover as part of the LEGO CITY Mars Research Shuttle (60226). The set is quite affordable, but about the only thing its little rover has resembling Curiosity rover are the number of wheels. Beyond that, there is no rocker-bogie and no corner steering, which is not a surprise at this price point.

Moving up the price and complexity scale is the LEGO NASA Mars Science Laboratory Curiosity Rover (21104), released in 2014 and no longer in production. This was part of the LEGO Ideas program where people can submit their LEGO designs and, if enough other LEGO fans voted in favor, the LEGO group will consider adapting it for production. This particular idea received over 10,000 supporting votes and became an official product. Perusing its assembly instruction PDF, I was surprised to find that this rover has a fully articulating rocker-bogie suspension. Very impressive! I also sadly learned that it required several pieces that I lack in my own LEGO collection, so I couldn’t “just” follow the directions to build one of my own. At some point I could go hunt for the few missing pieces which should be cheaper than buying this out-of-production kit on eBay. People are asking over $100 for a used kit and as much as $1,100 for a never-opened box. That’s more than two Sawppy rovers!

But if price is no object, I can look to other LEGO creations people have posted. There are quite a few “LEGO rover” projects online, and it follows roughly the same trajectory as the past few window shopping posts: most don’t even have six wheels, some have six wheel rocker-bogie but no corner steering, etc.

An example of a rover with corner steering is this LEGO MINDSTORM NXT creation. Despite being constrained by the selection of available LEGO pieces, it nevertheless replicated some features that I skipped with my 3D-printed rover. One example being the spokes in each of the six wheels, which I had simplified for 3D printing but the builder here faithfully represented more of the curves. And they have a robot arm, which my Sawppy is still waiting for. But according to the text, only the four corner wheels drive leaving the middle wheels free-wheeling. And it’s not obvious if the rocker-bogie articulates faithfully. At the very minimum, this design is missing the differential bar across the top.

This rover looks to be roughly the same size as my Sawppy rover, but the price tag would be significantly higher. There are six visible NXT brains on this invention: five on board the rover and one for the handheld remote. Each $350 MINDSTORM NXT box comes with a single NXT brain, so this rover costs at a minimum 6 * $350 = $2,100 dollars. Yikes.

So as far as LEGO rovers go, the best ones that implement features I consider important are build-it-yourself designs by enthusiasts and not available as commercial products. Thus we return to the world of 3D printing, where Sawppy isn’t the only rover design floating around.

Window Shopping 4tronix M.A.R.S. Rover

Wheeled robot kits calling themselves rovers are plentiful, but rarely do they faithfully represent the JPL rovers sent to Mars. ServoCity’s Bogie Runt Rover at least is a six-wheel chassis with rocker-bogie suspension, but is missing the corner wheel steering system. Looking for commercially available products out there with six wheel drive rocker-bogie and four corner steering, I was almost ready to conclude none existed until I came across the 4tronix M.A.R.S Rover Robot.

The name is an acronym for Mobile Autonomous Robotic System and the autonomy comes from the brainpower of either a Raspberry Pi Zero or BBC micro:bit. Out of all the commercial products I’ve come across, M.A.R.S. has the most mechanically faithful model of Curiosity and Perseverance suspension geometry. It has roughly similar proportions, and uses a differential arm over the body. Sojourner, Spirit, and Opportunity use a differential inside the body in order to maximize top surface area for solar panels. The nuclear-powered Curiosity and Perseverance didn’t have such constraint. By moving the differential above, they gained body interior volume.

Perusing the assembly instructions I see the structural components are all printed circuit boards (PCB) which is not usually considered structural material but apparently can be made to work at this scale. Six wheel drive comes from N20 micro gear motors, and four wheel steering comes from MG90S metal gear micro servos. Both of these are fairly widely available for maker projects and good choices.

My quibble with the M.A.R.S. rover are with its rotational motion joints. First up are the rocker-bogie suspension components, each of the joints are a M3 screw inside a PCB hole. I’m queasy about using fasteners as axles, since their threads are not really designed to carry a load. There are no bearings on these joints and the screws have to be tightened precisely. Too tight, and the joints would bind. Too loose, and the rover will wobble. This seems awfully finicky and would loosen as the rover travels.

Second, each wheel’s weight is transferred directly into the N20 micro gearbox. Examining pictures of these gearboxes online, it does not appear the gearbox is designed to handle loads perpendicular to rotational axis. Perhaps the rover is lightweight enough the load would be fine, but all I see are metal shafts turning inside holes in brass plate.

And third, each of the corner steering wheel assemblies are attached directly to the steering micro servo’s horn. Despite the metal gears, MG90S type servos still has a great deal of flexibility and play in its gearbox in the face of loads perpendicular to rotational axis. We can see each corner steering assembly wobbling noticeably while in motion:

(I also smiled when I realized this video cuts out right when the rover starts tackling a very challenging vertical climb. We’ll never know if it managed the climb.)

The final quibble I have with this design is that the steering axis is not aligned with the wheel. In less formal terms, it means the wheel is sideways offset from its corresponding steering servo instead of sitting directly under the servo. As a result, when the steering servo rotates, its wheel isn’t pivoting about a point. Instead, it is dragged through an arc. This adds a great deal of mechanical stress to the steering mechanism. It also makes the desired wheel velocity harder to calculate through a turn, but it appears the M.A.R.S. software skips that detail. All left side wheels are commanded to turn at the same rate, and all right side wheels are the same with each other. As a result we can see a bit of wheel skid through a turn, reintroducing some of the problems of using a strictly skid steer system as the Bogie Runt Rover did.

In conclusion M.A.R.S. has the mechanical sophistication of corner steering in addition to a good representation of rocker-bogie suspension, but its current iteration does not yet take full advantage. Still, it looks like a nifty little kit and the price of one hundred pounds is significantly lower than the cost of parts for Sawppy.

This concludes my quick tour of commercially available rover kits. But before I move on to DIY rovers, there’s a rover category that bridges across both “commercial product” and “do it yourself”: LEGO rovers.

Window Shopping ServoCity Bogie Runt Rover

The littleBits Space Rover is a cute little thing, but with only two wheels it bears little resemblance to real life Martian robot explorers. Moving a little bit higher on the fidelity scale, there are some robots out there with six wheels arranged in a rocker-bogie geometry. A representative of this breed is the ServoCity Bogie Runt rover.

Browsing the ServoCity web site, it appears that the Bogie Runt is the largest and most complex (so far?) of their line of “Runt Rover” chassis kits. Peewee, Sprout, and Zip Runt Rovers are two-wheel differential drive chassis like the littleBits Space Rover. Then there are the Junior, Half-Pint, and Whippersnapper Runt Rovers which roll on four wheels. The Bogie Runt is the only one with six wheels, and the only one with an articulating suspension system.

On ServoCity’s “Robot Chassis Kits” page they are all listed under “Smooth Surface.” This is certainly true for all the other Runt Rovers, but the Bogie Runt should be able to handle rougher terrain thanks to the rocker-bogie suspension system. It’s not obvious if the smooth surface classification for Bogie Runt was a mistake, or if it was deliberate based on information not immediately visible from just window shopping.

All of the Runt Rover kits appear to be made by laser (or possibly waterjet) cutting of flat materials, likely some type of plastic. They have bolt patterns to connect with ServoCity’s Actobotics build system which is mostly aluminum. Actobotics is very popular with FIRST Robotics participants and because of that, it was also the build system selected for the JPL Open Source Rover.

But that also causes a problem I have with the Bogie Runt: Actobotics is based on Inch dimensions and thus problematic for use outside of the United States. Sawppy is metric and, for worldwide compatibility, I plan to keep all derivatives and descendants metric.

Another shortfall of Bogie Runt is the lack of corner steering mechanisms. It is thus constrained to differential drive (a.k.a. “Tank Steer”) which works very well for two wheels but precision drops as the number of contact patches go up. In my experience, differential drive is marginal for four wheels, and behaves unpredictably with six. It was a tradeoff the Bogie Runt Rover product made against cost and complexity, leaving room for improvement and motivation for me to keep looking.

Window Shopping littleBits Space Rover Inventor Kit

Before I start getting too ambitious with making my personal rover project more like a user-friendly commercial product, I should look around to see if a product already existed. There are a lot of robot kits out there trying to earn that STEM education money, and some of them do try to get a slice of the excitement around our interplanetary robotic explorers. And while some of them are interesting and have their own merits, most of them lack features that I consider critical in a rover relative.

A representative example is the Space Rover Inventor Kit from littleBits. I like the idea of littleBits, giving beginners a very friendly way to explore electronics by making components with magnetic connectors. This makes electrical connections easy and also helps avoid beginner mistakes like accidentally reversing electrical polarity. The downside is that the customer has to pay a premium for this capability, and it’s not a premium that I’ve been personally willing to pay. This particular kit has a MSRP of $200 USD, and for that kind of money I would much rather build my own robot. (Though at time of this posting, the kit is on sale for $30.)

As a supplement to generic electronics beginner kits, littleBits also apply their technology to some themed offerings like this Space Rover kit. All the electrical components are reusable and remixable littleBits centered around a robot control module. The module is interesting because it interacts with a phone app, giving the rover remote control capabilities. But I also see the app as a disadvantage: as I understand it, the app would discourage experimentation. If the user changes the rover around, it wouldn’t take much before it exceeds what the phone app can gracefully handle. Thus motivating the user to leave it in the space rover configuration, defeating the point of using littleBits to begin with.

But in my eyes the biggest shortfall of this rover is its two-wheeled differential drive chassis. Similar to what home robot vacuums like Roomba uses. This is fine for flat indoor ground, which is what earthbound rovers mostly end up doing anyway. So as a home education toy this is a good technical choice. But for fans of rocker-bogie like myself, this is sorely lacking and we must look elsewhere.

But at least this little Space Rover has an arm. Looks like it only has one degree of freedom, which is very limiting, but that’s one more degree of freedom than Sawppy’s nonexistent robot arm. Victory: littleBits Space Rover!

Window Shopping: NASA Perseverance Rover 3D Print Static Model

After finishing a model of Mars rover Curiosity, the obvious question is: did NASA release a 3D print static model of its successor Perseverance? And the answer is yes! Curiously, the web site page points to a 3D file suitable for graphics rendering, not for 3D printing. But poking around the GitHub repository revealed there’s a “M2020” 3D print model. (Before receiving its name, Perseverance was called Mars 2020.)

Armed with my recent experience, I looked over the files for this printable rover. First the good news: the rocker-bogie suspension is represented in much higher fidelity on this model. The full rocker bogie geometry is represented, including a differential bar that is designed for some bent paperclips to serve as critical linkages. The wheel tracks appear to be correct with the center pair of wheels having wider tracks than the front and rear pairs. And the geometry is more accurate, no weird right angle bends as concession to ease of printing.

The lack of concession to ease of printing is also the bad news. Unlike the previous model, none of the geometry has been modified to fit flat on a print bed. Printing this model will require, at a minimum, printing with supports which is always problematic. Dual-material printer with dissolvable supports should make such designs easy to print, but I don’t have one of those.

Another change from the previous model is that this one doesn’t use snap-together construction. Parts are designed to be glued together instead. It also means these files assume a much higher printing precision, since superglue requires a much tighter fit than snap-together construction.

If I saw these traits on a Thingiverse item, I would be skeptical that the model is even buildable. That site is littered with too many things that are obviously impossible, merely the dream someone created in CAD and never test printed.

But this one appears to be real, since among the STL files is a picture of this rover design that has been built. Looking over the print quality of its parts, it was obviously printed at high detail quality on a good printer. Likely better than mine! I think I’ll hold off printing this rover design for the moment. Maybe later, when I have a well dialed-in printer that I can trust to meet precise tolerance requirements. In the meantime, I can admire the Perseverance 3D Model released by NASA that lives strictly in the digital realm.

Seeed Studio Odyssey X86J4105 Has Good ROS2 Potential

If I were to experiment with upgrading my Sawppy to ROS2 right now, with what I have on hand, I would start by putting Ubuntu ARM64 on a Raspberry Pi 3 for a quick “Hello World”. However, I would also expect to quickly run into limitations of a Pi 3. If I wanted to buy myself a little more headroom, what would I do?

The Pi 4 is an obvious step up from the 3, but if I’m going to spend money, the Seeed Studio Odyssey X86J4105 is a very promising candidate. Unlike the Pi, it has an Intel Celeron processor on board so I can build x86_64 binaries on my desktop machine and copy them straight over. Something I hope to eventually be a painless option for ROS2 cross compilation to ARM, but we’re not there yet.

This board is larger than a Raspberry Pi, but still well within Sawppy’s carrying capacity. It’s also very interesting that they copied the GPIO pin layout from Raspberry Pi, the idea some HATs can just plug right in is very enticing. Although that’s not a capability that would be immediately useful for Sawppy specifically.

The onboard Arduino co-processor is only useful for this application if it can fit within a ROS2 ecosystem, and the good news is that it is based on the SAMD21. Which makes it powerful enough to run micro-ROS, an option not available to the old school ATmega32U4 on the LattePanda boards.

And finally, the electrical power supply requirements are very robot friendly. The spec sheet lists DC input voltage requirement at 12V-19V, implying we can just put 4S LiPo power straight into the barrel jack and onboard voltage regulators will do the rest.

The combination of computing power, I/O, and power flexibility makes this board even more interesting than an Up Board. Definitely something to keep in mind for Sawppy contemplation and maybe I’ll click “Add to Cart” on this nifty little board (*) sometime in the near future.

(*) Disclosure: As an Amazon Associate I earn from qualifying purchases.

Update on ARM64: ROS2 on Pi

When I last looked at running ROS on a Raspberry Pi robot brain, I noticed Ubuntu now releases images for Raspberry Pi in both 32-bit and 64-bit flavors but I didn’t know of any compelling reason to move to 64-bit. The situation has now changed, especially if considering a move to the future of ROS2.

The update came courtesy of an announcement on ROS Discourse notifying the community that supporting 32-bit ARM builds have become troublesome, and furthermore, telemetry indicated that very few ROS2 robot builders were using 32-bit anyway. Thus the support for that platform is demoted to tier 3 for the current release Foxy Fitzroy.

This was made official on REP 2000 ROS 2 Releases and Target Platforms showing arm32 as a Tier 3 platform. As per that document, tier 3 means:

Tier 3 platforms are those for which community reports indicate that the release is functional. The development team does not run the unit test suite or perform any other tests on platforms in Tier 3. Installation instructions should be available and up-to-date in order for a platform to be listed in this category. Community members may provide assistance with these platforms.

Looking at the history of ROS 2 releases, we can see 64-bit has always been the focus. The first release Ardent Apalone (December 2017) only supported amd64 and arm64. Support for arm32 was only added a year ago for Dashing Diademata (May 2019) and only at tier 2. They kept it at tier 2 for another release Eloquent Elusor (November 2019) but now it is getting dropped to tier 3.

Another contributing factor is the release of Raspberry Pi 4 with 8GB of memory. It exceeded the 4GB limit of 32-bit addressing. This was accompanied by an update to the official Raspberry Pi operating system, renamed from Raspbian to Raspberry Pi OS, is still 32-bit but with mechanisms to allow addressing 8GB of RAM across the operating system even though individual processes are limited to 3GB. The real way forward is to move to a 64-bit operating system, and there’s a beta 64-bit build of Raspberry Pi OS.

Or we can go straight to Ubuntu’s release of 64-bit operating system for Raspberry Pi.

And the final note on ROS2: a bunch of new tutorials have been posted! The barrier for transitioning to ROS2 is continually getting dismantled, one brick at a time. And it’s even getting some new attention in long problematic areas like cross-compilation.

Window Shopping ARCore: API Documentation

Investigating Google ARCore for potential robotics usage, it was useful to review their Fundamental Concepts and Design Guidelines because it tells us the motivation behind various details and the priorities of the project. That gives us context around what we see in the nuts and bolts of the actual APIs.

But the APIs are where “the rubber meets the road” and where we leave all the ambitions and desires behind: the actual APIs implemented in shipping phones define the limitations of reality.

We get a dose of reality pretty immediately: estimation of phone pose in the world comes with basically no guarantees on global consistency.

World Coordinate Space
As ARCore’s understanding of the environment changes, it adjusts its model of the world to keep things consistent. When this happens, the numerical location (coordinates) of the camera and Anchors can change significantly to maintain appropriate relative positions of the physical locations they represent.

These changes mean that every frame should be considered to be in a completely unique world coordinate space. The numerical coordinates of anchors and the camera should never be used outside the rendering frame during which they were retrieved. If a position needs to be considered beyond the scope of a single rendering frame, either an anchor should be created or a position relative to a nearby existing anchor should be used.

Since it is on a per-frame basis, we could get Pose and PointCloud from a Frame. And based on that text, these would then need to be translated through anchors somehow? The first line of Anchor page makes it sound that way:

Describes a fixed location and orientation in the real world. To stay at a fixed location in physical space, the numerical description of this position will update as ARCore’s understanding of the space improves.

However, I saw no way to retrieve any kind of identifying context for these points. Ideally I would want “Put an anchor on that distinctive corner of the table” or some such. But still, “Working with anchors” has basic information on how it is useful. But as covered in many points throughout ARCore documentation, use of anchors must be kept at a minimum due to computational expense. Each Anchor is placed relative to a Trackable, and there are many ways to get one. The biggest hammer seems to be getAllTrackables from Sesson, which has a shortcut of createAnchor. There are more narrowly scoped ways to query for Trackable points depending on scenario.

Given what I see of ARCore APIs right now, I’m still a huge fan of future potential. Unfortunately its current state is not a slam dunk for robotics application, and that is not likely to change in the near future due to explicit priorities set by the product team.

But while I had my head buried in studying ARCore documentation, another approach popped up on the radar: the OpenCV AI Kit.

Window Shopping Google ARCore: Design Guidelines

After I got up to speed on fundamental concepts of Google ARCore SDK, I moved on to their design recommendations. There are two parts to their design guidelines: an user experience focused document, and a software development focused variant. They cover many of the same points, but from slightly different perspectives.

Augmented reality is fascinating because it has the potential to create some very immersive interactive experiences. The downside is that an user may get so immersed in the interaction they lose track of their surroundings. Much of the design document described things to avoid in an AR app that boiled down to: please don’t let the user hurt themselves. Many potential problems were illustrated by animated cartoon characters, like this one of an user walking backwards so focused on their phone they trip over an obstacle. Hence one of the recommendations is to avoid making users walk backwards.

Image source: Google

Some of the user experience guidelines help designers avoid weaknesses in ARCore capabilities. Like an admission that vertical surfaces can be challenging, because they usually have fewer identifiable features as compared to floors and tabletops. I found this interesting because some of the advertised capabilities, such as augmented images, are primarily targeted to vertical surfaces yet it isn’t something they’ve really figured out yet.

What I found most interesting was the discouragement of haptic feedback in both the UX design document and the developer document. Phone haptic feedback are usually implemented as a small electric motor spinning an unbalanced weight, causing vibration. This harms both parts of Structure from Motion calculations at the heart of phone AR: vibration adds noise to the IMU (inertial measurement unit) tracking motion, and vibration blurs the video captured by the camera.

From a robotics adaption viewpoint, this is discouraging. A robot chassis will have motors and their inevitable vibrations, some of which would be passed on to a phone bolted to the chassis. The characteristics of this vibration noise would be different from shaky human hands, and given priority of the ARCore team they would work to damp out the human shakiness but robotic motions would not be a priority.

These tidbits of information have been very illuminating, leading to the next step: find more details in the nitty gritty API documentation.

Window Shopping Google ARCore: Tracking

I started learning about Google ARCore SDK by reading the “Fundamental Concepts” document. I’m not in it for augmented reality, but to see if I can adapt machine vision capabilities for robotics. So while there are some interesting things ARCore could tell me about a particular point in time, the real power is when things start moving and ARCore works to track them.

The Trackable object types in the SDK represent data points in three dimension space that ARCore, well, tracks. I’m inferring these are visible features that are unique enough in the visible scene to be picked out across multiple video frames, and whose movement across those frames were sufficient for ARCore to calculate its position. Since those conditions won’t always hold, individual points of interest will come and go as the user moves around in the environment.

From there we can infer such an ephemeral nature would require a fair amount of work to make such data useful for augmented reality apps. We’d need to follow multiple feature points so that we can tolerate individual disappearances without losing our reference. And when new interesting features comes on to the scene, we’d need to decide if they should be added to the set of information followed. Thankfully, the SDK offers the Anchor object to encapsulate this type of work in a form usable by app authors, letting us designate a particular trackable point as important, and telling ARCore it needs to put in extra effort to make sure that point does not disappear. This anchor designation apparently brings in a lot of extra processing, because ARCore can only support a limited number of simultaneous anchors and there are repeated reminders to release anchors if they are no longer necessary.

So anchors are a limited but valuable resource for tracking specific points of interest within an environment, and that led to the even more interesting possibilities opened up by ARCore Cloud Anchor API. This is one of Google’s cloud services, remembering an anchor in general enough terms that another user on another phone can recognize the same point in real world space. In robot navigation terms, it means multiple different robots can share a set of navigation landmarks, which would be a fascinating feature if it can be adapted to serve as such.

In the meantime, I move on to the ARCore Design Guidelines document.

Window Shopping Google ARCore: Concepts

I thought Google’s ARCore SDK offered interesting capabilities for robots. So even though the SDK team is explicitly not considering robotics applications, I wanted to take a look.

The obvious starting point is ARCore’s “Fundamental Concepts” document. Here we can confirm the theory operation is consistent with an application of Structure from Motion algorithms. Out of all the possible type of information that can be extracted via SfM, a subset is exposed to applications using the ARCore SDK.

Under “Environmental Understanding” we see the foundation supporting AR applications: an understanding of the phone’s position in the world, and of surfaces that AR objects can interact with. ARCore picks out horizontal surfaces (tables, floor) upon which an AR object can be placed, or vertical surfaces (walls) upon which AR images can be hung like a picture. All other features build on top of this basic foundation, which also feel useful for robotics: most robots only navigate on horizontal surfaces, and try to avoid vertical walls. Knowing where they are relative to current position in the world would help collision detection.

The depth map is a new feature that caught my attention in the first place, used for object occlusion. There is also light estimation, helping to shade objects to fit in with their surroundings. Both of these allow a more realistic rendering of a virtual object in real space. While the depth map has obvious application for collision detection and avoidance more useful than merely detecting vertical wall surfaces. Light estimation isn’t obviously useful for a robot, but maybe interesting ideas will pop up later.

In order for users to interact with AR objects, the SDK includes the ability to map the user’s touch coordinate in 2D space into the corresponding location in 3D space. I have a vague feeling it might be useful for a robot to know where a particular point in view is in 3D space, but again no immediate application comes to mind.

ARCore also offers “Augmented Images” that can overlay 3D objects on top of 2D markers. One example offered: “for instance, they could point their phone’s camera at a movie poster and have a character pop out and enact a scene.” I don’t see this as a useful capability in a robotics application.

But as interesting as these capabilities are, they are focused on a static snapshot of a single point in time. Things get even more interesting once we are on the move and correlate data across multiple points in space or even more exciting, multiple devices.

Robotic Applications for “Structure From Motion” and ARCore

I was interested to explore if I can adapt capabilities of augmented reality on mobile device to an entirely different problem domain: robot sensing. First I had to do a little study to verify it (or more specifically, the Structure from Motion algorithms underneath) isn’t fundamentally incompatible with robots in some way. Once I gained some confidence I’m not barking up the wrong tree, a quick search online using keywords like “ROS SfM” returned several resources for applying SfM to robotics including several built on OpenCV. A fairly consistent theme is that such calculations are very computationally intensive. I found that curious, because such traits are inconsistent with the fact they run on cell phone CPUs for ARCore and ARKit. A side trip explored whether these calculations were assisted by specialized hardware like “AI Neural Coprocessor” that phone manufacturers like to tout on their spec sheet, but I decided that was unlikely for two reasons. (1) If deep learning algorithms are at play here, I should be able to find something about doing this fast on the Google AIY Vision kit, Google Coral dev board, or NVIDIA Jetson but I came up empty-handed. (2) ARCore can run on some fairly low-frills mid range phones like my Moto X4.

Finding a way to do SFM from a cell phone class processor would be useful, because that means we can potentially put it on a Raspberry Pi, the darling of hobbyist robotics. Even better if I can leverage neural net hardware like those listed above, but that’s not required. So far my searches have been empty but something might turn up later.

Turning focus back to ARCore, a search for previous work applying ARCore to robotics returned a few hits. The first hit is the most discouraging: ARCore for Robotics is explicitly not a goal for Google and the issue closed without resolution.

But that didn’t prevent a few people from trying:

  • An Indoor Navigation Robot Using Augmented Reality by Corotan, et al. is a paper on doing exactly this. Unfortunately, it’s locked behind IEEE paywall. The Semantic Scholar page at least lets me sees the figures and tables, where I can see a few tantalizing details that just make me want to find this paper even more.
  • Indoor Navigation Using AR Technology (PDF) by Kumar et al. is not about robot but human navigation, making it less applicable for my interest. Their project used ARCore to implement an indoor navigation aid, but it required the environment to be known and already scanned into a 3D point cloud. It mentions the Corotan paper above as part of “Literature Survey”, sadly none of the other papers in that section was specific to ARCore.
  • Localization of a Robotic Platform using ARCore (PDF) sounded great but, when I brought it up, I was disappointed to find it was a school project assignment and not results.

I wish I could bring up that first paper, I think it would be informative. But even without that guide, I can start looking over the ARCore SDK itself.

Augmented Reality Built On “Structure From Motion”

When learning about a new piece of technology in a domain I don’t know much about, I like to do a little background research to understand the fundamentals. This is not just for idle curiosity: understanding theoretical constraints could save a lot of grief down the line if that knowledge spares me from trying to do something that looked reasonable at the time but is actually fundamentally impossible. (Don’t laugh, this has happened more than once.)

For the current generation of augmented reality technology that can run on cell phones and tablets, the fundamental area of research is “Structure from Motion“. Motion is right in the name, and that key component explains how a depth map can be calculated from just a 2D camera image. A cell phone does not have a distance sensor like Kinect’s infrared projector/camera combination, but it does have motion sensors. Phones and tablets started out with only a crude low resolution accelerometer for detecting orientation, but that’s no longer the case thanks to rapid advancements in mobile electronics. Recent devices have high resolution, high speed sensors that integrate accelerometer, gyroscope, and compass across X, Y, and Z axis. These 9-DOF sensors (3 types of data * 3 axis = 9 Degrees of Freedom) allow the phone to accurately detect motion. And given motion data, an algorithm can correlate movement against camera video feed to extract parallax motion. That then feeds into code which builds a digital representation of the structure of the phone’s physical surroundings.

Their method of operation would also explain how such technology could not replace a Kinect sensor, which is designed to sit on the fireplace mantle and watch game players jump around in the living room. Because the Kinect sensor bar does not move, there is no motion from which to calculate structure making SfM useless for such tasks. This educational side quest has thus accomplished the “understand what’s fundamentally impossible” item I mentioned earlier.

But mounted on a mobile robot moving around in its environment? That should have no fundamental incompatibilities with SfM, and might be applicable.

Google ARCore Depth Map Caught My Attention

Once I decided to look over an augmented reality SDK with an intent for robotics applications, I went to look at Google’s ARCore instead of Apple’s ARKit for a few reasons. The first is hardware: I have been using Android phones so I have several pieces of ARCore compatible hardware on hand. I also have access to computers that I might be able to draft into Android development duty. In contrast, Apple ARKit development requires MacOS desktop machines and iOS hardware, which is more expensive and rare in my circles.

The second reason was their announcement that ARCore now has a Depth API. Their announcement included two animated GIFs that caught my immediate attention. The first shows that they can generate a depth map, with color corresponding to distance from camera.

ARCore depth map
Image source: Google

This is the kind of data I had previously seen from a Xbox 360 Kinect sensor bar, except the Kinect used an infrared beam projector and infrared camera to construct that depth information on top of its RGB camera. In comparison, Google’s demo implies that they can derive similar information from just a RGB camera. And given such a depth map, it should be theoretically possible to use it in a similar fashion to a Kinect. Except now the sensor would be far smaller, battery powered, and works in bright sunlight unlike the Kinect.

ARCore occlusion
Image source: Google

Here is that data used in ARCore context: letting augmented reality objects be properly occluded by obstacles in the real world. I found this clip comforting because its slight imperfections assured me this is live data of a new technology, and not a Photoshop rendering of what they hope to accomplish.

It’s always the first question we need to ask of anything we see on the internet: is it real? The depth map animation isn’t detailed enough for me to see if it’s too perfect to be true. But the occlusion demo is definitely not too perfect: there are flaws in the object occlusion as the concrete wall moved in and out of the line of sight between us and the animated robot. This is most apparent in the second half of the clip, as the concrete wall retreated we could see bits of stair that should have been covered up by the robot but is still visible because the depth map hadn’t caught on yet.

Incomplete occlusion

So this looks nifty, but what was the math magic that made it possible?

Might A Robot Utilize Google ARCore?

Machine vision is a big field, because there are a lot of useful things we can do when a computer understands what it sees. In narrow machine-friendly niches it has become commonplace, for example the UPC bar code on everyday merchandise is something created for machines to read, and a bar code reader is a very simplified and specific niche of machine vision.

But that is a long, long way from a robot understanding its environment through cameras, with many sub sections along the path which are entire topics in their own right. Again we have successes in narrow machine-friendly domains such as a factory floor set up for automation. Outside of environments tailored for machines, it gets progressively harder. Roomba and similar robot home vacuums like Neato could wander through a human home, but their success depends on a neat and tidy spacious home. As a home becomes more cluttered, success rate of robot vacuums decline.

But they’re still using specialized sensors and not a camera with vision comparable to human sight. Computers have no problems chugging through a 2D array of pixel data, but extracting useful information is hard. The recent breakthrough in deep learning algorithms opened up more frontiers. The typical example is a classifier, and it’s one of the demos that shipped with Google AIY Vision kit. (Though not the default, which was the “Joy Detector.”) With a classifier the computer can say “that’s a cat” which is a useful step toward something a robot needs, which is more like “there’s a house pet in my path and I need to maneuver around it, and I also need to be aware it might get up and move.” (This is a very advanced level of thinking for a robot…)

The skill to pick out relevant physical structure from camera image is useful for robots, but not exclusively to robots. Both Google and Apple are building augmented reality (AR) features into phones and tablets. Underlying that feature is some level of ability to determine structure from image, in order to overlay an AR object over the real world. Maybe that capability can be used for a robot? Time for some research.

Window Shopping Firmata: Connect Microcontrollers To Computers

Reading about LabVIEW LINX stirred up memory of something with a similar premise. I had forgotten its name and it took a bit of research to re-discover Firmata. Like LINX, Firmata is a protocol for communicating between microcontrollers and desktop computers. Like LINX, there are a few prebuilt implementations available for direct download, such as their standard Arduino implementation of Firmata.

There’s one aspect of the Firmata protocol I found interesting: its relationship to MIDI messages. I had originally thought it was merely inspired by MIDI messages, but the Firmata protocol documentation says it is actually a proper subset of MIDI. This means Firmata messages have to option to coexist with MIDI messages on the same channel, conveying data that is mysterious to MIDI instruments but well-formed to not cause problems. This was an interesting assertion, even with the disclaimer that in practice Firmata typically runs at a higher serial speed on its own bus.

Like LINX, Firmata is intended to be easily implemented by simple hardware. The standard Arduino implementation can be customized for specific projects, and anything else that can communicate over a serial port is a candidate hardware endpoint for Firmata.

But on the computer side, Firmata is very much unlike LINX in its wide range of potential software interfaces. LINX is a part of LabVIEW, and that’s the end of the LINX story. Firmata can be implemented by anything that can communicate over a serial port, which should cover almost anything.

Firmata’s own Github hosts some Python sample code, and it is but one of five options for Python client libraries listed on the protocol web site and they carry along some useful tips like using Python’s ord()/chr() to convert hexadecimal data to/from Firmata packets. Beyond Python, every programming language I know of are invited to the Firmata party: Processing, Ruby, JavaScript, etc.

Since I had been playing with C# and .NET recently, I took a quick glance at their section of Firmata. These are older than UWP and use older non-async APIs. The first one on the list used System.IO.Ports.SerialPort, and needed some workaround for Mono. The second one isn’t even C#: It’s aimed at Visual Basic. I haven’t looked at the third one on the list.

If I wanted to write an UWP application that controls hardware via Firmata, writing a client library with the newer async Windows.Devices.SerialCommunication.SerialDevice API might be a fun project.

Windows Shopping LINX: Connecting LabVIEW To Maker Hardware

When I looked over LabVIEW earlier with the eyes of a maker, my biggest stumbling block was trying to connect to the kind of hardware a maker would play with. LabVIEW has a huge library of hardware interface components for sophisticated professional level electronics instrumentation. But I found nothing for simple one-off projects like the kind I have on my workbench. I got as far as finding a reference to a mystical “Direct I/O” mechanism, but no details.

In hindsight, that was perfectly reasonable. I was browsing LabVIEW information presented on their primary site targeted to electronics professionals. I thought the lack of maker-friendly information meant National Instruments didn’t care about makers, but I was wrong. It actually meant I was not looking at the right place. LabVIEW’s maker-friendly face is on an entirely different site, the LabVIEW MakerHub.

Here I learned about LINX, an architecture to interface with maker level hardware starting with the ubiquitous Arduino, Raspberry Pi, and extensible to others. From the LINX FAQ and the How LINX Works page I got the impression it allows individual LabVIEW VI (virtual instrument) to correspond to individual pieces of functionality on an Arduino. But very importantly, it implies that representation is distinct from the physical transport layer, where there’s only one serial (or WiFi, or Ethernet) connection between the computer running LabVIEW and the microcontroller.

If my interpretation is true this is a very powerful mechanism. It allows the bulk of LabVIEW program to be set up without worrying about underlying implementation. Here’s one example that came to mind: A project can start small with a single Arduino handling all hardware interface. Then as the project grows, and the serial link becomes saturated, functions can be split off into separate Arduinos with their own serial link plugged in to the computer. Yet doing so would not change the LabVIEW program.

That design makes LabVIEW much more interesting. What dampens my enthusiasm is the lack of evidence of active maintenance on LabVIEW MakerHub. I see support for BeagleBone Black, but not any of the newer BeagleBone boards (Pocket is the obvious candidate.) The list of supported devices list Raspberry Pi only up to 2, Teensy only up to 3.1, Espressif ESP8266 but not the ESP32, etc. Balancing that discouraging sight is that the code is on Github, and we see more recent traffic there as well as the MakerHub forums. So it’s probably not dead?

LINX looks very useful when the intent is to interface with LabVIEW on the computer side. But when we want something on the computer other than LabVIEW, we can use Firmata which is another implementation of the concept.

UPDATE: And just after I found it (a few years after it launched) NI is killing MakerHub with big bold red text across the top of the site: “This site will be deprecated on August 1, 2020″

Window Shopping: Progressive Web App

When I wrote up my quick notes on ElectronJS, I had the nagging feeling I forgot something. A few days later I remembered: I forgot about Progressive Web Apps (PWA), created by some people at Google who agrees with ElectronJS that their underlying Chromium engine can make a pretty good host for local offline applications.

But even though PWA and ElectronJS share a lot in common, I don’t see them as direct competitors. What I’ve seen on ElectronJS is focused on creating applications in the classic sense. They are primarily local apps, just built using technologies that were born in the web world. Google’s PWA demos showcase extension of online web sites, where the primary focus is on the web site but PWA lets them have a local offline supplement.

Given that interpretation, a computer control panel for an electronics hardware project is better suited to ElectronJS than a PWA. At least, as long as the hardware’s task is standalone and independent of others. If a piece of hardware is tied to a network of other similar or complementary pieces, then the network aspect may favor a PWA interfacing with the hardware via Web USB. Google publishes a tutorial showing how to talk to a BBC micro:bit using a Chrome serial port API. I’m not yet familiar with the various APIs to know if this tutorial used the web standard or if it uses the Chrome proprietary predecessor to the standard, but its last updated date of 2020/2/27 implies the latter.

Since PWA started as a Google initiative, they’ve enabled it in as many places as they could starting with their own platforms like Android and ChromeOS. They are also supported via Chrome browser on major desktop operating systems. The big gap in support are Apple’s iOS platforms, where Apple forbids a native code Chrome browser and more generally any application platforms. There are some technical reasons but the biggest hurdle is financial: installing a PWA bypasses Apple’s iOS app store, a huge source of revenue for the company, so Apple has a financial disincentive.

In addition to Google’s PWA support via Chrome, Microsoft supports PWA on Windows via their Edge browser with both the old EdgeHTML and new Chromium-based versions, though with different API feature levels. While there’s a version of Edge browser for Xbox One, I saw no mention of installing PWAs on an Xbox like a standard title.

PWAs would be worth a look for network-centric projects that also have some offline capabilities, as long as iOS support is not critical.