Phoebe Digital Avatar in RViz

Now that Phoebe URDF has been figured out, it has been added to RViz visualization of Phoebe during GMapping runs. Before this point, Phoebe’s position and orientation (called a ‘pose‘ in ROS) is represented by a red arrow on the map. It’s been sufficient to get us this far, but a generic arrow is not enough for proper navigation because it doesn’t represent the space occupied by Phoebe. Now, with the URDF, the volume of space occupied by Phoebe is also visually represented on the map.

This is important for a human operator to gauge whether Phoebe can fit in certain spaces. While I was driving Phoebe around manually, it was a guessing game whether the red arrow will fit through a gap. Now with Phoebe’s digital avatar in the map, it’s a lot easier to gauge clearance.

I’m not sure if the ROS navigation stack will use Phoebe’s URDF in the same way. The primary reason the navigation tutorial pointed me to URDF is to get Phoebe’s transforms published properly in the tf tree using the robot state publisher tool. It’s pretty clear robot footprint information will be important for robot navigation for the same reason it was useful to human operation, I just don’t know if it’s the URDF doing that work or if I’ll end up defining robot footprint some other way. (UPDATE: I’ve since learned that, for the purposes of ROS navigation, robot footprint is defined some other way.)

In the meantime, here’s Phoebe by my favorite door to use for distance reference and calibration.

Phoebe By Door Posing Like URDF

And here’s the RViz plot, showing a digital representation of Phoebe by the door, showing the following:

  • LIDAR data in the form of a line of rainbow colored dots, drawn at the height of the Neato LIDAR unit. Each dot represents a LIDAR reading, with color representing the intensity of each return signal.
  • Black blocks on the occupancy map, representing space occupied by the door. Drawn at Z height of zero representing ground.
  • Light gray on the occupancy map representing unoccupied space.
  • Dark gray on the occupancy map representing unexplored space.

Phoebe By Door

(Cross-posted to Hackaday.io)

Next Phoebe Project Goal: ROS Navigation

rosorg-logo1When I started working on my own TurtleBot variant (before I even decided to call it Phoebe) my intention was to build a hardware platform to get first hand experience with ROS fundamentals. Phoebe’s Hackaday.io project page subtitle declared itself as a ROS robot for <$250 capable of SLAM. Now that Phoebe can map surroundings using standard ROS SLAM library ‘gmapping‘, that goal has been satisfied. What’s next?

One disappointment I found with existing ROS SLAM libraries is that the tutorials I’ve seen (such as this and this) expect a human to drive the robot during mapping. I had incorrectly assumed the robot would autonomously exploring its space, but “simultaneous location and mapping” only promises location and mapping – nothing about deciding which areas to map, and how to go about it. That is left to the human operator.

When I played with SLAM code earlier, I decided against driving the robot manually and instead invoked an existing module that takes a random walk through available space. A search on ROS Answers web site for something more sophisticated than a random walk resulted in multiple pointers to the explore module, but that code hasn’t been maintained since ROS “groovy” four versions ago. So one path forward is to take up the challenge of either update explore or write my own explorer.

That might be interesting, but once a map is built, what do we do with it? The standard ROS answer is the robot navigation stack. This collection of modules is what gives a ROS robot the ability to plan a path through a map, watch its progress through that plan, and update the plan in reaction to unexpected elements in the environment.

At the moment I believe it would be best to learn about the standard navigation stack and getting that up and running on Phoebe. I might return to the map exploration problem later, and if so, seeing how map data is used for navigation will give me better insights into what would make a better map explorer.

(Cross-posted to Hackaday.io)

Robot Disorientation Traced To Timing Mismatch

Once the Roboclaw ROS Node‘s wheel parameters were updated to match the new faster motors, Phoebe went on a mapping run. And the results were terrible! This is not a complete surprise. Based on previous experimentation with this LIDAR module, I’ve seen its output data distorted by motion. It doesn’t take a lot of motion – even a normal human walking pace is enough to cause visible distortion. So now that Phoebe’s motors are ten times faster than before, that extra speed also adds extra distortion.

While this is a problem, it might not be the only problem behind the poor map. I decided to dig into an anomaly I noticed while using RViz to calibrate wheel data against LIDAR data: there’s some movement that can’t be entirely explained by a LIDAR spinning at 240 RPM.

The idea behind this odometry vs. LIDAR plot on RViz is to see if wheel odometry data agrees with LIDAR data. My favorite calibration surface is a door – it is a nice flat surface for LIDAR signal return, and I could swing the door at various angles for testing. In theory, when everything lines up, movement in the calculated odometry would match LIDAR observed behavior, and everything that is static in the real world (like the door) would be static in the plot as well.

In order to tune the base_width parameter, I looked at the position of the door before turning, and position after turning. Adjusting base_width until they line up, indicating odometry matches LIDAR. But during the turn, the door moved relative to the robot before finishing at the expected position.

When Phoebe started turning (represented by red arrow) the door jumped out of position. After Phoebe stopped turning, the door snapped back to position. This means non-moving objects appear to the robot as moving objects, which would confuse any mapping algorithm.

Odom LIDAR Mismatch No Hack

I chased down a few dead ends before realizing the problem is a timing issue: the timestamp on the LIDAR data messages don’t line up with the timestamp on odometry messages. At the moment I don’t have enough data to tell who is at fault, the LIDAR node or the Roboclaw node, just that there is a time difference. Experimentation indicated the timing error is roughly on the order of 100 milliseconds.

Continuing the experiment, I hard-coded a 100ms timestamp delta. This is only a hack which does not address the underlying problem. With this modification, the door still moves around but at least it doesn’t jump out of place as much.

Odom LIDAR Mismatch 100ms hack

This timing error went unnoticed when Phoebe was crawling extremely slowly. But at Phoebe’s higher speed this timing error could not longer be ignored. Ideally all of the objects on the RViz plot would stay still, but we clearly see nonuniform distortion during motion. There may be room for further improvement, but I don’t expect I’ll ever get to ideal performance with such an inexpensive LIDAR unit. Phoebe may end up just having to go slowly while mapping.

(Cross-posted to Hackaday.io)

Using RViz to Validate Motor Movement Against LIDAR Data

The Roboclaw ROS node is responsible for calculating odometry information based on encoder values read from each wheel. In order to translate them into standard ROS units, it needs two parameters:

  • ticks_per_meter : To calculate physical distance traversed by each wheel, the code needs to know how many encoder counts it takes for the wheels to travel one meter.
  • base_width : To calculate how much the robot has turned, the code needs to know how far apart the two drive wheels are placed.

Both of these values needed updating after upgrading Phoebe to the second chassis. I got impatient with the slow speed of the first draft, so the motor gearboxes were changed out with ones that deliver less torque and precision but much faster top speed. This change of gearing would need a new ticks_per_meter value. And the second chassis is slightly wider than the first, which obviously changes the base_width value. Both of these could be calculated on paper, but that is only a starting point. Real world is always a little different from the theoretical and needs a little adjustment.

The easiest place to start is ticks_per_meter. Phoebe is placed on a flat surface, next to a ruler, and commanded to drive straight forward for a short distance. During this activity, the odometry data is monitored with rostopic echo /odom to see how far Phoebe thinks it has actually gone. If the ruler said Phoebe didn’t go as far as it thought it did, increase ticks_per_meter. If Phoebe overshot, reduce that value.

Once wheel travel was verified with a ruler, LIDAR is added to the picture. RViz is commanded to plot odometry data and LIDAR data together, and Phoebe is placed facing a door serving as a large flat surface for reference. The red arrow represents where Phoebe thinks it is, facing the horizontal line representing its LIDAR’s view of the door.

Phoebe Door Test 1 Start

Then Phoebe was commanded to move backwards 0.5m. If odometry data agrees with LIDAR data, the movement away from door and the door’s distance from robot would match, canceling each other out so the line representing the door would not move on the RViz plot. It looks like the ruler calibration worked out well, as we’re only a tiny bit off.

Phoebe Door Test 2 Back 0.5m.png

Once distance was verified, we move on to rotation. Command Phoebe to make a 90 degree turn clockwise and see if LIDAR plot agrees. Again, ideally the turn calculated from odometry would agree with the LIDAR plot, leaving the door in roughly the same place on the RViz plot. Ideally.

Phoebe Door Test 3 90 Deg Right Bad

In this case, however, the door shows a minor clockwise rotation. This change of position in LIDAR data indicates Phoebe didn’t turn as far as it thought it did. To adjust parameters so Phoebe’s calculations better align with actual motion, we can increase the base_width parameter. And if the door had rotated the other way (Phoebe turned further than it thought it did) the parameter should be decreased.

(Cross-posted to Hackaday.io)

Phoebe vs. Office Chair Round 2

Phoebe was built to roam my house, but the first draft chassis was unable to do so effectively due to a few problems that the second chassis aimed to solve. The first one was ground clearance, which was solved by raising the main chassis and sloping the bottom of the electronics tray. Sloping that leading edge gives Phoebe a better approach angle for smoothly transitioning between floor surfaces.

The second major problem was the LIDAR scanner’s height: it was too high to see the legs of an office chair. Hence the other major goal of the second chassis was to lower the LIDAR mounting point and hopefully bring an office chair’s legs into plane of view.

Placing the newly rebuilt Phoebe next to the chair looks promising at first glance. Unlike the taller first chassis, the LIDAR’s horizontal plane of sight is now low enough it should be able to see the legs.

Phoebe vs Chair Round 2

The proof is in the occupancy grid, and the RViz plot shows that Phoebe can now see the legs of the chair blocking its way.

Phoebe Sees Chair Legs

It’s not a very solid detection, though. Something about the surface texture and/or angle of the plastic results in a weak laser return.  And there’s the risk of a leg going undetected when if approached from the end, as the dark sloped rounded end of the chair leg is nearly invisible to LIDAR.

But it’s a huge improvement from before, where the LIDAR was too high to see any part of the starfish pattern. It’s good enough for us to proceed with the next task: integrate Phoebe’s new faster wheel drive motors into the system.

(Cross-posted to Hackaday.io)

ROS Notes: Map Resolution

Now that I’m driving my TurtleBot-derived robot Phoebe around and mapping my house, I’m starting to get some first-hand experience with robotic mapping. One of the most fascinating bits of revelation concerns map resolution.

When a robot launches the Gmapping module, one of the parameters (delta) dictates the granularity of the occupancy grid in meters. For example, setting it to 0.05 (the value used in TurtleBot 3 mapping demo) means each square in the grid is 0.05 meters or 5 cm on each side.

This feels reasonable for a robot that roams around a household. Large objects like walls would be fine, and the smallest common obstacle in a house like table legs can reasonably fill a 5cm x 5cm cell on the occupancy grid. If the grid cells were any larger, it would have trouble properly accounting for chair and table legs.

Low Resolution Sharp Map 5cm

So if we make the grid cells smaller, we would get better maps, right?

It’s actually not that simple.

The first issue stems from computation load. Increasing resolution drastically increases the amount of memory consumed to track the occupancy grid, and increases computation requirements to keep grid cells updated. The increase in memory consumption is easy to calculate. If we halve the grid granularity from 5cm to 2.5cm, that turns each 5cm square into four 2.5cm squares. Quadrupling the memory requirement for our occupancy grid. Tracking and maintaining this map is a lot more work. In my experience the mapping module has a lot harder time matching LIDAR scan data to the map, causing occasional skips in data processing that ends up reducing map quality.

The second issue stems from sensor precision. An inexpensive LIDAR like my unit salvaged from a Neato robot vacuum isn’t terribly precise, returning noisy distance readings that varies over time even if the robot and the obstacle are both standing still. When the noise exceeds map granularity, the occupancy grid starts getting “fuzzy”. For example, a solid wall might no longer be a single surface, but several nearby surfaces.

High Resolution Fuzzy Map 1cm

As a result of those two factors, arbitrarily increasing the occupancy map resolution can drastically increase the cost without returning a worthwhile improvement. This is a downside to going “too small”, and it was less obvious than the downside of going “too large.” There’s a “just right” point in between that makes the best trade-offs. Finding the right map granularity to match robots and their tasks is going to be an interesting challenge.

(Cross-posted to Hackaday.io)

Phoebe The Cartographer

Once odometry calculation math in the Roboclaw ROS driver was fixed, I could drive Phoebe (my TurtleBot variant) around the house and watch laser and odometry data plotted in RViz. It is exciting to see the data stream starting to resemble that of a real functioning autonomous robot! And just like all real robots… the real world does not match the ideal world. Our specific problem of the day is odometry drift: Phoebe’s wheel encoders are not perfectly accurate. Whether from wheel slippage, small debris on the ground, or whatever else, they cause the reported distance to be slightly different from actual distance traveled. These small errors accumulate over time, so the position calculated from odometry becomes less and less accurate as Phoebe drives.

The solution to odometry drift is to supplement encoder data with other sensors, using additional information can help correct for position drift. In the case of Phoebe and her  TurtleBot 3 inspiration, that comes in courtesy of the scanning LIDAR. If Phoebe can track LIDAR readings over time and build up a map, that information can also be used to locate Phoebe on the map. This class of algorithms is called SLAM for Simultaneous Location and Mapping. And because they’re fundamentally similar robots, it would be straightforward to translate TurtleBot 3’s SLAM demo to my Phoebe.

There are several different SLAM implementations available as ROS modules. I’ll start with Gmapping because that’s what TurtleBot 3 demo used. As input this module needs LIDAR data in the form of ROS topic /scan and also the transform tree published via /tf, where it finds the geometry relationship between odometry (which I just fixed), base, and laser. As output, it will generate an “occupancy grid”, a big table representing a robot’s environment in terms of open space, obstacle, or unknown. And most importantly for our purposes: it will generate a transform mapping map coordinate frame to the odom coordinate frame. This coordinate transform is the correction factor to be applied on top of odometry-calculated position, generated by comparing LIDAR data to the map.

Once all the pieces are in place, Phoebe can start mapping out its environment and also correct for small errors in odometry position as it drifts.

SLAM achievement: Unlocked!

Phoebe GMapping

(Cross-posted to Hackaday.io)

Roboclaw ROS Driver: Odometry Calculation Reversal

rosorg-logo1I’m making good progress on integrating ROS into my TurtleBot variant called Phoebe. I’ve just configured RViz to plot Phoebe’s location as reported by Roboclaw ROS driver‘s odometry information topic /odom. On the same plot, I’ve also placed data reported by Phoebe’s scanning LIDAR. Everything looked good when I commanded Phoebe to drive forward, but the geometry went askew as soon as Phoebe started turning. The LIDAR dots rotated one way, and the odometry reporting arrow rotated another. Something is wrong in the geometry calculation code.

For this particular problem I knew exactly where to start looking. When I assembled Phoebe, I knew I needed to configure motors in the way expected by the Roboclaw ROS driver. The README didn’t explicitly say which motor went on which side, so I went into the source code and got conflicting answers. In the motor driving routine (cmd_vel_callback) it performed calculation for right wheel motion and sent it to motor 1, and left wheel motion went to motor 2.

However, this doesn’t match encoder calculation code. The method EncodeOdom::update_publish expects its parameters to be (enc_left, enc_right). In method Node::run, it retrieves encoder values from motor 1 and saves it in variable enc1, and enc2 for motor 2, then calls update_publish(enc1, enc2). Which would treat encoder value of motor 1 as enc_left instead of enc_right, the opposite of what we’d want.

So the fact data on /odom is going the wrong way was not a surprise, but why didn’t the LIDAR transform also suffer the same problem? This was traced down to an earlier change submitted as a fix for the reverse transform problem – that commit added a negative sign in front of the calculated angle when broadcasting odom -> base_link transform. But that only masked and didn’t fix the underlying problem. The transform might look right, but /odom data is still all wrong, and variables like self.last_enc_right is actually holding the left side value.

The correct fix would be to reverse the parameter order when calling EncodeOdom::update_publish which correctly assigns encoder 1 count the right motor, and encoder 2 the left motor. And with the underlying problem fixed, we no longer need the negative sign so that can be deleted.

With this fix, Phoebe’s laser plot and odometry plot in RViz appear to agree with each other and both correspond correctly to the direction Phoebe is turning. I’ve submitted this fix as a pull request on the Roboclaw ROS driver.

Driving Miss Phoebe (Not Self-Driving… Yet)

Now that the Roboclaw ROS node is configured for Phoebe TurtleBot’s physical layout, and the code is running reliably instead of crashing, I can drive it around as a remote control car. This isn’t the end goal, of course. If all I wanted was a remote control car there were far cheaper and easier ways to get here. Right now the goal is just to verify functionality of the drive system and that it is properly integrated into the ROS infrastructure.

In order to send driving commands to Phoebe, I need something that publishes to the command & velocity topic /cmd_vel. On an autonomous robot, the navigation code would be in charge of sending these commands. But since it’s a pretty common task to move a robot around manually, there are standard ROS packages to drive a robot via human commands. The most basic is teleop_twist_keyboard which allows driving by pressing keys on a keyboard. Alternatively there is teleop_twist_joy for driving via a joystick.

Those remote control (or ‘tele operation’ in ROS parlance) nodes worked right off the bat, which is great. But I quickly got bored with driving Phoebe around on the carpet in front of me. First I launched RViz to visualize scanning LIDAR data as I did before. This was enough for me to drive Phoebe beyond my line of sight, watching surroundings in the form of the laser scan dots. After I verified that still worked, I stepped up the difficulty: I wanted RViz to plot laser data on top of odometry data in order to verify that Roboclaw ROS node is generating odometry data correctly.

To do this, I needed to participate in the ROS coordinate transform stack, the infrastructure to track all the frames of reference for relating robot components to each other in physical space. The Roboclaw ROS node publishes a transform to relate a robot’s position reference frame base_link to the odometry origin reference frame odom. The LIDAR ROS node publishes its data relative to its own neato_laser reference frame. As the robot builder, it was my job to write a transform to relate neato_laser frame to Phoebe’s base_link frame. Fortunately, ROS transform tutorials covered this exact task and I quickly got my desired RViz plot.

RViz with LaserScan and Odometry

It looks like the LIDAR scan plot from earlier, but now there’s an arrow indicating Phoebe’s position and direction. The bigger change not visible in this screen shot is that RViz is now plotting in the odometry frame. This means we no longer watch strictly from Phoebe’s viewpoint  where robot stays in the center of screen. The plot is now in odometry frame, and Phoebe should be moving relative to the map.

I drove Phoebe forward, and was happy to see the laser scan stayed still relative to the map and the red arrow representing Phoebe moved forward. But when I started turning Phoebe, the red arrow turned one way and the LIDAR plot moved the opposite way.

Something is wrong in the coordinate transform data, time for some debugging…

Roboclaw ROS Driver: Add Thread Synchronization

rosorg-logo1I now turn my attention to finding the root cause of random sporadic failures when the Roboclaw ROS driver makes calls into Roboclaw API. The most common symptom of these failures is the TypeError I addressed earlier, but avoiding crash by TypeError is only a band-aid. There were still other issues. Sometimes Phoebe’s movement stutters. And even more mysteriously, sometimes when Phoebe was supposed to be standing still, it would twitch for a fraction of a second.

Looking into ROS logs, the node never intended to send a movement. But it does send a continuous stream of “run at speed X” commands at a rate of ten per second, whether it is trying to move the robot or not. When staying still, that stream continues at the same rate constantly sending “run at speed zero”. The fact that my robot twitches when it’s supposed to be standing still tells me that “run at speed zero” command is occasionally corrupted into a “run at speed X” command. Which starts the motor moving for 1/10th of a second until it is stopped by the next non-corrupted “run at speed zero” command.

Any time there is a random sporadic failure, my instinct is to look for places where threading collisions may be taking place. Programming for multi-threaded environments can get tricky. When I wrote code for SGVHAK Rover, the intent was to make that code easy to understand and pick up. In that spirit, I explicitly kept everything on a single thread to avoid multi-threading issues.

But now we’re playing in the major leagues and there’s no avoiding multiple threads. ROS itself runs across multiple processes and threads, so it could scale to robots that have multiple on board computers, each of which has multiple processing cores. Fortunately, ROS implementation takes care of almost everything, each ROS node just has to make sure they’re doing the right thing with their own private data.

There are at least three different ROS topics involved in this Roboclaw ROS node:

  1. Driving commands received via /cmd_vel
  2. Odometry computation broadcast to /odom
  3. Diagnostics information

Each ROS topic is processed in own thread, so given three topics, we should expect at least three different threads who might all call into Roboclaw API at the same time to perform their tasks. Armed with this knowledge, I looked for code managing cross-thread access. My plan was to review that code to see if I can find any obvious problems with it.

That plan was changed when I found no code managing cross-thread access. I guess its absence qualifies as an obvious problem and it would certainly explain the kind of behavior I saw.

Not being a Python expert, I cruised StackOverflow for a pattern I could use to implement Python thread synchronization. I decided it was most straightforward to use the with keyword described on one of the later replies on this “Semaphores on Python” thread. Using this pattern makes the code change delta very straightforward to read.

There were a few initial calls into Roboclaw API to set things up, I left those alone. But as soon as the code started kicking off events that would have other threads (specifically, when it started the diagnostics thread) every following call into Roboclaw API is synchronized by a threading.Lock() object. With this modification, we can guarantee that only one thread will be performing serial communication to Roboclaw motor controller at any given time, and avoid data corruption by multiple threads trying to talk to the serial port at the same time.

Phoebe ran smoothly and reliably after this work. No more stutter in motion, and no more twitching when standing still. I’ve submitted the fix as a pull request.

Roboclaw ROS Driver: Encoder Count Logging Error

rosorg-logo1Once I fixed the ticks_per_meter error, I could drive Phoebe around by sending movement command messages to the /cmd_vel topic. However, I could not do this for very long (sometimes minutes, sometimes mere seconds) before the Roboclaw ROS driver would crash with the following error:

File "roboclaw_node.py", line 232, in run
    rospy.logdebug(" Encoders %d %d" % (enc1, enc2))
TypeError: %d format: a number is required, not NoneType

Once crashed, Phoebe no longer responds to movement commands, and that is bad. What’s going on here? The instance of TypeError that brought the whole thing crashing down gives us a place to start: one of its %d parameters (which could have been either enc1 or enc2) has a value of None instead of a number representing encoder count.

Looking through the source code file, I could see that the encoder values were initialized to None then replaced with an encoder count by a call to Roboclaw API ReadEncM1 / ReadEncM2. If this call should fail, there’s a try/except block to catch the error, but that leaves the encoder value at None, causing the crash I’ve experienced.

There exists an if statement whose intention might be to avoid this crash by checking for existence of the variable, but I don’t understand how it would help. Encoder count variables would always exist since they were declared and initialized to None.

There are two parts to this problem:

  1. Why is the Roboclaw API call failing?
  2. If it fails, how can we avoid crashing on TypeError?

Part 2 is easier so let’s address that first. TypeError complains when the variables are not numbers, so let’s explicitly check for numbers. Looking through trusty StackOverflow found this method of checking for numeric values in Python, so let’s use that to guard against TypeError.

This change would not address the root cause of API call failing, it’s just a band-aid to avoid fatally crashing and burning. When we fail to retrieve encoder count using Roboclaw API, our internal information become stale which causes more problems down the line. Problems like bad odometry calculations, which will cascade to more problems.

What we really need to do is to address the failing API call, which I’ll work on next. In the meantime this band-aid has been submitted via a pull request.

Roboclaw ROS Driver: Encoder Ticks Per Meter Of Travel

rosorg-logo1Now that Phoebe has taken physical form (imperfect as a first draft tends to be) it’s time to put together software necessary to run it. We’ve already previously established how to use the Neato LIDAR in ROS, so now focus turns to propulsion control via Roboclaw motor controllers. On their product information page, Roboclaw manufacturer linked to this Github repository with the label “Robot Operating System Driver”. So obviously that’s where I should start.

I had to look through this code earlier to learn “Motor 1” of the Roboclaw should be the right hand motor and “Motor 2” the left, but up to this point I’ve only ran this code enough to verify the correct motors turn in the correct direction. Now that I have to configure the driver parameter details to match my physical robot.

A critical parameter is ticks_per_meter, the number of encoder counts that transpire when the robot travels a meter. This encapsulates multiple pieces of information about a particular mechanical implementation: things like motor gear ratio and wheel diameter. This count is required for accurate velocity control, since velocity is specified in meters per second. It is also obviously important to calculate robot odometry, because Roboclaw tracks distance in encoder counts and ROS expects position information in meters.

To test this, I send a command “travel forward at 0.1 meters per second” for one second of time. Configuration is successful when this command makes Phoebe travel 10 centimeters, with corresponding confirmation in output odometry calculations. First I ran the test with default parameters, which resulted in a little over 1 centimeter of travel implying my robot’s ticks_per_meter is almost ten times that of default.

I then re-ran the test with a much higher value for ticks_per_meter parameter and the travel distance didn’t change. Then I tried even larger values, and smaller values, and no matter what the robot kept going forward the same amount. After I double checked that I was spelling my parameter correctly, I went looking into the code.

Thankfully the author had included a line to output parameter to ROS debug log system. (rospy.logdebug("ticks_per_meter %f", self.TICKS_PER_METER)) Once I found this debug message in the log, I was able to confirm that the default value was always used no matter what value I passed in.

Backtracking through the code, I found the cause: The call to retrieve parameter value has a typo, so it is looking for parameter tick_per_meter (singular tick) instead of the intended ticks_per_meter (plural ticks.) Once I added the missing ‘s‘, Phoebe started moving the correct distance. Once verified I submitted this fix via a pull request.

Establish Motor Directions For Phoebe TurtleBot

The first revision of Phoebe’s body frame has mounting points for the two drive wheels and the caster wheel. There are two larger holes to accommodate drive motor wiring bundle, and four smaller holes to mount a battery tray beneath the frame. Since this is the first rough draft, I didn’t bother spending too much time over thinking further details. We’ll wing it and take notes along the way for the next revision.

Phoebe Frame First Draft.PNG

After the wheels were installed, there was much happiness because the top surface of the frame sat level with the ground, indicating the height compensation (for height difference between motorized wheels and caster in the back) was correct or at least close enough.

Next, two holes were drilled to mechanically mount the Roboclaw motor control module. Once secured, a small battery was connected plus both motor drive power wires. Encoder data wires were not connected, just taped out of the way, as they were not yet needed for the first test: direction of motor rotation.

Phoebe TurtleBot Stage 1 PWM

The Roboclaw ROS node expects the robot’s right side motor to be connected as Motor #1, and the left as Motor #2. It also expects positive direction on both motors to correspond to forward motion.

I verified robot wiring using Ion Studio, the Windows-based utility published by the makers of Roboclaw. I used Ion Studio to command the motors via USB cable to verify the right motor rotates clockwise for positive motion, and the left motor counter-clockwise for positive motion. I got it right on the first try purely by accident, but it wouldn’t have been a big deal if one or both motors spun the wrong way. All I would have had to do is to swap the motor drive power wires to reverse their polarity.

(Cross-posted to Hackaday.io)

New Project: Phoebe TurtleBot

While my understanding of the open source Robot Operating System is still far from complete, I feel like I’m far enough along to build a robot to run ROS. Doing so would help cement the concepts covered so far and serve as an anchor to help explore new areas in the future.

Sawppy Rover is standing by, and my long-term plan has always been to make Sawppy smarter with ROS. Sawppy’s six-wheeled rocker-bogie suspension system is great for driving over rough terrain but fully exploiting that capability autonomously requires ROS software complexity far beyond what I could handle right now. Sawppy is still the goal for my ROS journey, but I need something simpler as an intermediate step.

A TurtleBot is the official ROS entry-level robot. It is far simpler than Sawppy with just two driven wheels and restricted to traveling over flat floors. I’ve been playing with a simulated TurtleBot 3 and it has been extremely helpful for learning ROS. Robotis will happily sell a complete TurtleBot 3 Burger for $550. This represents a discounted bundle price less than the sum of MSRP for all of its individual components, but $550 is still a nontrivial chunk of change. Looking over its capabilities, though, I’m optimistic I could implement critical TurtleBot functionality using parts already on hand.

Components for parts bin turtlebot phoebe

  • Onboard computer: This one is easy. I have several Raspberry Pi 3 sitting around I could draft into the task, with all necessary accessories like a microSD card, a case, and power supply.
  • Laser scanner: Instead of the Robotis LDS-01, I’ll use a scanning LIDAR salvaged from a Neato robot vacuum that I bought off eBay for experimentation. Somebody has already written a driver package to report its data as a ROS /scan topic and I’ve already verified it sends data that the ROS visualization tool RViz can understand.
  • Motor controller: Robotis OpenCR is a very capable board with lots of expansion capabilities, but I have a RoboClaw left from SGVHAK Rover (JPL Open Source Rover) project so I’ll use that instead. Somebody has already written a driver package to accept commands via ROS topic /cmd_vel and report motion via /odom, though I haven’t tried it yet.
  • Gearmotor + Encoder: TurtleBot 3 Burger’s wheels are driven by Robotis Dynamixel XL430-W250-T serial bus servos via their OpenCR board. Encoders are critical for executing motor commands sent to /cmd_vel correctly and for reporting odometry information via /odom. Fortunately some gearmotor+encoder from Pololu are available. These were formerly SGVHAK Rover’s steering motors, but after one of them broke they were all replaced with beefier motors. I’m going to borrow two of the remaining functioning units for this project.
  • Wheel: Robotis has a wheel designed to bolt onto their servos, and they have mounting hardware for their servos. I’m going to borrow the wheel and mounting hardware from a self-balancing robot toy gathering dust on a shelf. (Similar but not identical to this one.) That toy didn’t have encoders on its motors, but they have the same mounting points and output shaft as the Pololu gearmotor so it was an easy swap. The third wheel, a free wheeling caster, was salvaged from a retired office chair.
  • Chassis hardware: Robotis has designed a modular system (like an Erector Set) for building robot chassis like their TurtleBot variants. As for me… I have a 3D printer and I’m not afraid to use it.
  • Battery: I’ll be borrowing Sawppy’s 5200 mAh battery pack.

This forms the roster for my TurtleBot variant, with an incremental component cost of $0 thanks to the parts bin. “Parts Bin TurtleBot” is a mouthful to say and not a very friendly name. I looked at its acronym “PB-TB” and remembered R2-D2’s nickname “Artoo”. So I’m going to turn “PB” into Phoebe.

I hereby announce the birth of my TurtleBot variant, Phoebe!

(Cross-posted to Hackaday.io)

Examining Basic Requirements For Mapping in ROS

When I first looked at Gazebo robot simulation environment, I didn’t understand how to find the interface layer difference between a simulated robot and a physical robot. Now I’ve learned enough ROS to know: Look at the ROS Node graph to find a node named /gazebo. Anything provided by or consumed by that node is a virtual substitute provided by Gazebo. When the same ROS code stack is on a physical robot, nodes that interface with physical hardware replaces everything provided in simulation by /gazebo.

When I tested a cheap little laptop’s performance in ROS, I put this knowledge to use. Gazebo ran on a high-end computer and every other node ran on the laptop. This simulates the workload of running Gmapping algorithm as if the low-end laptop was mounted on a physical robot. But what, specifically, is required? Let’s look at the ROS node graph once again with rqt_graph. Here’s a graph generated while TurtleBot 3’s mapping demo is running in Gazebo:

TurtleBot 3 GMapping With UI

Here’s a slightly different graph, generated by running the same mapping task but with Gazebo’s GUI and ROS RViz visualization tool turned off. They are useful for the human developer but are not strictly necessary for the robot to run. We see the /gazebo_gui node has dropped out as expected, and the /map topic was also dropped because it was no longer being consumed by RViz for presentation.

TurtleBot 3 GMapping No UI

We can see the Gazebo-specific parts are quite modest in this particular exercise. A physical robot running Gmapping in ROS will need to subscribe to /cmd_vel so it can be told where to go, and provide laser distance scanning data via /scan so Gmapping can tell where it is. Gazebo also publishes the simulated robot’s state via /tf and /joint_states.

And now that I have a spinning LIDAR working in ROS to provide /scan, the next project is to build a robot chassis that can be driven via /cmd_vel. After that is complete, we can use it to learn about /tf and /joint_states.

ROS Is Not News But Shows Up On Hacker News Anyway

ROS is now over a decade old and quite established in its niche. Every day, there are newbies like myself who start their journey to learn the ropes. Motivated by ambition to build software stacks for their own robot projects. Personally, I’ve known about ROS for a long time, though only now am I putting in the effort to work with it firsthand.

When it comes to big software frameworks I definitely count myself as “Grumpy Old Man”. I’ve seen them before, I’ll get proficient at it, and then I’ll move on to the next thing. I don’t get excited… it’s just a tool. But some ROS newcomers get really excited about their discovery! It may not be news in the historical sense, but it’s certainly news to them and they might choose to submit to venues like Reddit or, in today’s example, YCombinator’s Hacker News. And apparently it’s only been about 10 months since the last time someone submitted ROS to Hacker News.

ROS on Hacker News

And since the internet is the internet, the comment thread is naturally filled with positive people espousing the benefits of ROS, thankful of an open source robot infrastructure available so no one has to reinvent the wheel.

Just kidding.

It’s the internet.

The comment thread is actually full of gripes from people critical of the cruft built up during the course of a 10+ year software project. Complaints about how the generalized infrastructure isn’t good enough for their specific needs. Complaints about how it doesn’t use [insert favorite pet technology thing here] without considering its history as of ten years ago. Was it mature? Dependable? Did it even exist back then?

Having worked on big software projects full of legacy code, I look at these “ROS people are idiots” complaints and I shake my head. I know that a software project as big as ROS would have made major decisions based on state of software not ten years ago (when ROS was released) but more like twelve or thirteen years ago when they started putting it together. Plus if they made early decisions based on proven track record, that track record would have stretched even further back.

Yeah, ROS is old. Which is why the Open Source Robotics Foundation, the organization supporting ROS, has focused most of its resources into ROS 2. A ground-up rewrite incorporating the lessons learned during all this time. I’m cautiously optimistic ROS2 will be all it’s promised to be, but we’ll just have to stay tuned to see how reality lines up.

In the meantime, ROS is here, it is mature, and while ROS 2 will be changing a lot of technical fundamentals, it will not change any of the underlying philosophies. So I’ll stay focused learning classic ROS. It has its values, no matter what internet comments say.

Symptoms Of A Computer Struggling To Perform ROS Mapping

Once my Dell Inspiron 11 3000 (3180) laptop had its factory installation of Windows safely saved away in a Windows system image backup, its meager 32GB eMMC storage was wiped clean for an installation of Ubuntu 16.04 and ROS Kinetic Kame. Installation was mostly uneventful, except its touchpad stopped working a few minutes after setup began, forcing me to complete setup using keyboard only. This issue seemed to be resolved after updating Ubuntu to latest packages, so it was only a minor annoyance on the way to answering my $130 question: is this meager AMD E2-9000e processor powerful enough to serve as a robot brain? The answer: Yes, but barely.

Dell 3180 SLAM

My test was to run a standard ROS package that performs SLAM (simultaneous location and mapping) using Robotis’ virtual TurtleBot 3 in Gazebo simulation. To reflect the workload of a robot brain running OpenSlam’s GMapping algorithm, I ran only mapping code on the laptop. My desktop computer handled the complex physics simulation and rendering of Gazebo in order to keep the two workloads separate. To give some context for this little laptop’s capabilities, the same mapping workload was run on two other systems for comparison. One faster, and one slower, than this little laptop.

Representing the high end is my desktop computer with an Intel Core i5-7600. It kept up with incoming sensor data effortlessly and matched them up to existing records. Here’s an output log excerpt on the way to generating a high quality map:

Average Scan Matching Score=313.501
neff= 93.949
Registering Scans:Done
update frame 243
update ld=0.0096975 ad=0.30291
Laser Pose= -1.44014 1.54989 -0.977552
m_count 177
Average Scan Matching Score=312.594
neff= 93.949
Registering Scans:Done
update frame 244
update ld=0.0278723 ad=0.896994
Laser Pose= -1.46775 1.55373 -1.87455
m_count 178
Average Scan Matching Score=309.104
neff= 92.9918
Registering Scans:Done
update frame 245
update ld=0.116176 ad=0.441108
Laser Pose= -1.40149 1.64916 -2.31565
m_count 179
Average Scan Matching Score=311.613
neff= 92.9897
Registering Scans:Done
update frame 246
update ld=0.23972 ad=2.51909e-05
Laser Pose= -1.23899 1.8254 -2.31568
m_count 180

There is one interesting observation, though: according to CPU utilization metrics, the ROS Node executing Gmapping consumed 100% of a single CPU core. I don’t know if this means the algorithm has more room for improvement if given a faster core, or if this just means the algorithm takes up as much CPU as it can grab regardless of workload.

On the other end of the performance spectrum is the low power ARM processor of a Raspberry Pi 3, and this SLAM code was too much for the little chip to handle. CPU utilization metrics also show 100% utilization of a single core, but it looks like sensor data avalanches in too quickly for a Pi to process and match up to existing map data. There were only rare successful matches in the sea of errors, as seen in this excerpt of output:

Scan Matching Failed, using odometry. Likelihood=-1430.25
lp:0.367574 2.18056 -1.47424
op:1.16607 1.6985 -3.09185
Scan Matching Failed, using odometry. Likelihood=-1183.3
lp:0.367574 2.18056 -1.47424
op:1.16607 1.6985 -3.09185
Scan Matching Failed, using odometry. Likelihood=-0.874547
lp:0.367574 2.18056 -1.47424
op:1.16607 1.6985 -3.09185
Scan Matching Failed, using odometry. Likelihood=-0.900994
lp:0.367574 2.18056 -1.47424
op:1.16607 1.6985 -3.09185
Average Scan Matching Score=208.949
neff= 55.0344
Registering Scans:Done
update frame 28
update ld=1.28116 ad=2.45138
Laser Pose= 2.08673 0.807569 0.739956
m_count 28
Scan Matching Failed, using odometry. Likelihood=-1376.72
lp:1.16607 1.6985 -3.09185
op:2.08673 0.807569 0.739956
Scan Matching Failed, using odometry. Likelihood=-707.376
lp:1.16607 1.6985 -3.09185
op:2.08673 0.807569 0.739956
Scan Matching Failed, using odometry. Likelihood=-653.378
lp:1.16607 1.6985 -3.09185
op:2.08673 0.807569 0.739956
Scan Matching Failed, using odometry. Likelihood=-116.341
lp:1.16607 1.6985 -3.09185
op:2.08673 0.807569 0.739956

So how did the budget laptop perform in comparison, on its AMD E2-9000e processor?

It was far better than the Raspberry Pi, delivering mostly successful scan matches, but I could see occasional failure messages indicating it was struggling to keep pace with a fire hose of data. Curiously, CPU utilization did not stay pegged to 100%. It sometimes dip as low as 80%, implying there’s another bottleneck in the system keeping the CPU from being fully fed with work. But it’s the results that matter most. A visual examination of the map it generated looks rougher than one generated by my desktop, but usable. Meaning the resulting map might be of “good enough” quality for a robot to use despite its occasional errors.

So the little machine didn’t ace the test, but it managed to squeak by with a passing grade of C+, maybe a B-. This is very encouraging news for performance delivered by a low-end chip. It means we can start experimenting with this inexpensive laptop for now, and we have lots of upgrade headroom in the future.

Windows 10 WSL Can Run ROS, With Firewall Caveat

To win developer acceptance, Microsoft added WSL (Windows Subsystem for Linux) to 64-bit editions of Windows 10. The original iteration was only advertised to support common command-line utilities like ‘git‘ that perform relatively simple operations. However, the product has been evolving since its initial release and has become increasingly more functional to run more complex Linux software.

Could Windows 10 WSL run ROS? According to this thread on ROS Answers, it didn’t start out that way. But blocking bugs were found and fixed over past months, and now it’s possible to run ROS inside WSL. I tried this and found this to mostly work, with a minor caveat on networking.

When bringing a ROS software stack online, there is the concept of a “ROS Master”. This process listens on TCP port 11311 and serves to orchestrate communication with other ROS Nodes. Every ROS Node needs to talk to ROS Master at least once on startup. Which meant port 11311 is the one probed by researchers looking for unsecured ROS robots that were inadvertently connected to public internet.

The default network firewall on a Windows 10 computer is Microsoft’s own Windows Defender Firewall. It has a good default of ignoring all incoming traffic, unless an application explicitly asks to open up ports. At the moment this integration does not exist, so software inside WSL opening ports wouldn’t open up those same ports on Windows firewall. When running ROS in WSL, this means incoming traffic on port 11311 are blocked which results in the following:

  • ROS Master running in WSL is accessible to ROS Nodes running on the same computer, because traffic on the same computer is unaffected by firewall.
  • ROS Master running on another computer is accessible to ROS Nodes running in WSL, because outbound traffic is not blocked by firewall.
  • ROS Master running in WSL is NOT accessible to ROS Nodes running on another computer, because inbound traffic is blocked by firewall.

ERROR: Unable to communicate with master!

If a developer wishes to run networking-aware software inside WSL, we’d have to go into Windows Defender Firewall and manually add a permission for network access. Ideally we can set up a rule to allow port 11311 only when we’re running a ROS Master within WSL. But such fine-grained control is not available. For now, the only option is to open a port with no limitation. It sounds like some improvements are on the way, but even then it will still require explicit developer action.

To open port 11311, we need to first get to Windows Defender section within control panel and select “Advanced Settings”

Windows Defender security center

Then we can create a new “Inbound Rule” to allow traffic on 11311.

Windows Defender Firewall control panel

Since this is not a fine-grained control over port 11311 access, it’s not a good idea to leave this rule active at all times. For best practice, enable this rule only when running a ROS Master in WSL and only when that master needs to work with ROS Nodes running on other computers.

 

 

 

Dell Inspiron 11 3000 (3180) As Robot Brain Candidate

Well, I should have seen this coming. Right after I wrote I wanted to be disciplined about buying hardware, that I wanted to wait until I know what I actually needed, a temptation rises to call for a change in plans. Now I have a Dell Inspiron 11 3000 (3180) on its way even though I don’t yet know if it’ll be a good ROS brain for Sawppy the Rover.

Dell Notebook Inspiron 11 3000 3180

The temptation was Dell’s Labor Day sale. This machine in its bare-bones configuration has a MSRP of $200 and can frequently be found on sale for $170-$180. To kick off their sale event, Dell made a number of them available for $130 and that was too much to resist.

This particular hardware chassis is also sold as a Dell Chromebook, so the hardware specs are roughly in line with the Chromebook comments in my previous post. We’ll start with the least exciting item: the heart is a low-end dual-core x86 CPU, an AMD E2-9000e that’s basically the bottom of the totem pole for Intel-compatible processors. But it is a relatively modern 64-bit chip enabling options (like WSL) not available on the 32-bit-only CPUs inside my Acer Aspire or Latitude X1.

The CPU is connected to 4GB of RAM, far more than the 1GB of a Raspberry Pi and hopefully a comfortable amount for sensor data processing. Main storage is listed as 32GB of eMMC flash memory which is better than a microSD card of a Pi, if only by a little. The more promising aspect of this chassis is the fact that it is also sometimes sold with a cheap spinning platter hard drive so the chassis can accommodate either type of storage as confirmed by the service manual. If I’m lucky (again), I might be able to swap it out with a standard solid state hard drive and put Ubuntu on it.

It has most of the peripherals expected of a modern laptop. Screen, keyboard, trackpad, and a webcam that might be repurposed for machine vision. The accessory that’s the most interesting for Sawppy is a USB 3 port necessary for a potential depth camera. As a 11″ laptop, it should easily fit within Sawppy’s equipment bay with its lid closed. The most annoying hardware tradeoff for its small size? This machine does not have a hard-wired Ethernet port, something even a Raspberry Pi has. I hope its on-board wireless networking is solid!

And lastly – while this computer has Chromebook-level hardware, this particular unit is sold with Windows 10 Home. Having the 64-bit edition installed from the factory should in theory allow Windows Subsystem for Linux. This way I have a backup option to run ROS even if I can’t replace the eMMC storage module with a SSD. (And not bold enough to outright destroy the Windows 10 installation on eMMC.)

Looking at the components in this package, this is a great deal: 4GB of DDR4 laptop memory is around $40 all on its own. A standalone license of Windows 10 Home has MSRP of $100. That puts us past the $130 price tag even before considering the rest of the laptop. If worse comes to worst, I could transfer the RAM module out to my Inspiron 15 for a memory boost.

But it shouldn’t come to that, I’m confident even if this machine proves to be insufficient as Sawppy’s ROS brain, the journey to that enlightenment will be instructive enough to be worth the cost.

The Spectrum of ROS Robot Brain Candidates

rosorg-logo1In the immediate future, my ROS experimentation will follow the TurtleBot 3 model: An on-board Raspberry Pi 3 will read sensor input and send motor output. It will communicate over wireless network to a PC (either desktop or laptop) who will process sensor data, evaluate potential actions, and plan actions to be sent back to RPi3 for execution.

This should scale well to pretty much any output mechanisms we have on the horizon, especially since we’re likely to offload real time control to dedicated co-processor modules like a Roboclaw or serial bus servos. They take care of direct motor control, leaving the RPi3 with very little to do beyond sending a short command.

Input will be a different story. We know simple bump sensors will be easy. And while the Neato scanning LIDAR isn’t taxing to a Pi, its limited sensing capability means we will eventually need better sensors that give more data. A depth-sensing camera sensor like an old Kinect or an Intel RealSense looks interesting, but they require USB 3 which is beyond a Raspberry Pi. And while a Pi can handle simple vision processing of a Duckiebot, doing something more complex will quickly outstrip its capabilities.

We won’t know for sure what the bottleneck would be until we start building some robots and run into limitations firsthand. Until we know, we risk wasting money on unnecessary capability. Still, it’s good to have information on a spectrum of candidates.

raspberry-pi-logoWe already have a starting point on the low end of the spectrum: Raspberry Pi. There are several other competitors in the single-board computer market, almost all of which claim to be more powerful than a Pi. But very few could match the mass volume pricing of a Pi or its software ecosystem. The last part is important because ROS runs best on something that has a port of Ubuntu, which is absent from many Pi competitors.

The first step beyond a Raspberry Pi 3, then, is probably going to be something cheap with a low-end Intel processor that can run Ubuntu. I had thought my collection of old PC hardware could step into this space, but an Acer Aspire 10 really doesn’t want to run Linux and the Dell Latitude X1 is just too old. Its CPU consumes far more power than modern chips while doing far less. And even worse than that, its spinning platter hard drive consumes more power than SSDs while being slower.

chromebook logoThis points to a modern Chromebook with x86 CPU, most of which could be convinced to run Ubuntu. The CPU won’t be anything exciting. They might even be slower than the RPi’s ARM chip for some tasks, but they’ll have a larger cache and connect to more memory. Chromebooks also tend to have more robust main storage (instead of a microSD card) plus keyboard, screen, and battery. Some of these will even have USB 3 ports for a complete package starting at around $200.

NVIDIA logoIf a Chromebook proves insufficient, it’ll likely be due to the low-end CPU. Where we go beyond that will depend on the nature of the work overloading the chip. If it’s in the arena of dedicated vision or other AI-related processing capabilities, we might move to something like NVIDIA’s Jetson which has dedicated processing hardware for specific tasks and could run Ubuntu. They’re not cheap, so we have to be sure we need the specialized capabilities before shelling out the dollars.

Intel logoFor less task-specific needs, the Intel NUC product line is a good candidate: compact little boxes that holds RAM, SSD, USB3 ports and available with a wide range of processors. From unexciting Chromebook-equivalent chips all the way up to Core i7 paired with a respectable AMD GPU. And unlike dedicated hardware like Jetson, an Intel NUC can be repurposed to a wide spectrum of other projects if not serving as robot brain. They’re quite capable little Swiss Army knives of computing.

But there’s one remaining scenario where a NUC might prove insufficient. And that’s when we need a powerful Intel CPU in conjunction with a powerful NVIDIA GPU. This means a high-end ‘gamer laptop’ at which point my Inspiron 15 7000 (7577) might end up sitting in Sawppy‘s equipment bay. This marks the high end of the spectrum and I hope I don’t end up there, because that’s going to be a very expensive trip.