Speedy Phoebe: Swapping Gearbox For 370 Motors

The first rough draft chassis for Phoebe worked well enough for me to understand some pretty serious problems with that design. Now that I have a better idea what I’m doing, it’s time to start working on a new chassis incorporating lessons learned. And since all the components will be taken apart anyway, it would be a good time to address another problem: Phoebe’s speed. More precisely, the lack thereof.

Phoebe’s motor + encoder + gearbox unit was borrowed from the retired parts bin for SGVHAK Rover. Since they were originally purchased to handle steering, priority was on precision and torque rather than speed. It worked well enough for Phoebe to move around, but their slow speed meant it took quite some time to map a room.

The motor mounts used for Phoebe’s first draft chassis were repurposed from a self-balancing robot toy, which had a similar motor coupled with a different ratio gearbox. That motor was not suitable for ROS work because there was no encoder on the motor, but perhaps we could swap its gearbox with the motor that does have an encoder.

Identical output shaft and mount

Here’s the “Before” picture: Self-balancing robot motor + gearbox on the left, former SGVHAK rover steering encoder + motor + gearbox on the right. The reason I was able to use the self balancing robot’s motor mount and wheel is because they both had the same output shaft diameter and mount points. While they have identical diameter, the steering gearbox is noticeably longer than the balancing robot gearbox.

Both Are 370 Motors

Both of these motors conform to a generic commodity form factor called “370”. A search for “370 motor” on Alibaba will find many companies making different motors with different options of voltage, speed, etc. All are physically similar in size. Maybe even identical? As for what “370” means… best guess is that it originally referred to overall length of the motor at 370 millimeters. It doesn’t specifically mean anything for remaining motor dimensions, but “370 motor” has probably become a de-facto standard. (Like 608 bearings.)

After a few screws were removed, both gearboxes were easily disassembled. We can see a few neat things: the plate mounting the gearbox to the motor had multiple holes to accommodate three different patterns. Either these gearboxes were designed to fit on multiple different motors, or some 370 motors are made with different bolt patterns than others.

Gearboxes Removed

Fortunately, both motors (one with encoder, one without) seem to have the same bolt pattern. And more importantly – the gear mounted on the motor output shaft seems to be identical as well! I don’t have to pull the gear off one shaft and mount it on another, which is great because that process tends to leave the gear weaker. With identical gears already mounted on the motor output shaft, I can literally bolt on the other gearbox and complete the swap.

After - Gearboxes swapped

Voila! The motor with encoder now has a different gear ratio that should allow Phoebe to run a lot faster. The slow one was advertised to be 227:1 ratio. I don’t have specification sheet for the fast gearbox, but turning one shaft and counting revolutions of the other indicates a roughly 20:1 ratio. So theoretically Phoebe’s top speed has been increased ten-fold. Would that be too fast and cause Phoebe to run out of control? Would it be unable to slow to a sufficiently low crawl speed for Phoebe to cautiously explore new worlds? We won’t know until we try!

(Cross-posted to Hackaday.io)

Phoebe’s Nemesis: Office Chair

A little real-world experience with Phoebe has revealed several problems with my first rough draft chassis design. The second problem on the list is Phoebe’s LIDAR height: It sits too high to detect certain obstacles like an office chair. In the picture below, Phoebe has entered a perilous situation without realizing it. The LIDAR’s height meant Phoebe only sees the chair’s center post, thinking it is a safe distance away, blissfully ignorant of chair legs that have completely blocked its path.

Phoebe Faces Office Chair

Here is the RViz plot of the situation, representing Phoebe’s perspective. A red arrow indicates Phoebe’s position and direction. Light gray represents cells in the map occupancy grid thought to be open space, and black are cells that are occupied by an obstacle. Office chair’s center post is represented by two black squares at the tip of the red arrow, and the chair’s legs are absent because Phoebe never saw them with LIDAR.

Phoebe and Office Chair Post

This presents obvious problems in collision avoidance as Phoebe can’t avoid chair legs that it can’t see. Mounting height for Phoebe’s LIDAR has to be lowered in order to detect office chair legs.

Now that I’ve seen this problem firsthand, I realize it would also be an issue for TurtleBot 3 Burger. It has a compact footprint, and its parts are built upwards. This meant it couldn’t see office chair legs, either. But that’s OK as long as the robot is constrained to environments where walls are vertical and tall, like the maze seen in TurtleBot 3 Navigation demo. Phoebe would work well in such constrained environments too, but I’m not interested in constrained environments. I want Phoebe to roam my house.

Which leads us to Waffle Pi, the other TurtleBot 3 model. it has a larger footprint than the Burger, but it is a squat shape allowing LIDAR to be mounted lower and still have a clear view all around the top of the robot.

So I need to raise the bottom of Phoebe for ground clearance, and also lower the top for LIDAR mount. If the LIDAR can be low enough to look just over the top of the wheels, that should be good enough to see an office chair’s legs. Will I find a way to fit all of Phoebe’s components into this reduced height range? That’s the challenge at hand.

(Cross-posted to Hackaday.io)

Phoebe’s Nemesis: Floor Transitions

Right now my TurtleBot-based robot Phoebe is running around on a very rough first draft of chassis design. It was put together literally in an afternoon in the interest of time. Just throw the parts together so we can see if the idea will even work. Well, it did! And I’m starting to find faults with the first draft chassis that I want to address on the next version for a much better thought-out design.

The first fault is the lack of ground clearance. When I switched my mentality from the rough terrain capable Sawppy rover to a flat ground TurtleBot like Phoebe, I didn’t think the latter would need very much ground clearance at all. As a result, Phoebe’ battery pack hung between the driving wheels and caster, with only a few millimeters of clearance between the bottom of the battery tray and the ground.

Phoebe Ground Clearance Flat

If I’m not climbing rocks, I asked myself, why would I need ground clearance?

Well, I’ve found my answer: my home has rooms with carpet, rooms with linoleum, and rooms with tile. The transition between these surfaces are not completely flat. They’re pretty trivial for a walking human being, but for poor little Phoebe they are huge obstacles. Driving across the doorway from carpet to linoleum would cause Phoebe to get stuck on its battery belly.

Phoebe Ground Clearance Threshold

“More ground clearance” is a goal for Phoebe’s next chassis.

(Cross-posted to Hackaday.io)

ROS Notes: Map Resolution

Now that I’m driving my TurtleBot-derived robot Phoebe around and mapping my house, I’m starting to get some first-hand experience with robotic mapping. One of the most fascinating bits of revelation concerns map resolution.

When a robot launches the Gmapping module, one of the parameters (delta) dictates the granularity of the occupancy grid in meters. For example, setting it to 0.05 (the value used in TurtleBot 3 mapping demo) means each square in the grid is 0.05 meters or 5 cm on each side.

This feels reasonable for a robot that roams around a household. Large objects like walls would be fine, and the smallest common obstacle in a house like table legs can reasonably fill a 5cm x 5cm cell on the occupancy grid. If the grid cells were any larger, it would have trouble properly accounting for chair and table legs.

Low Resolution Sharp Map 5cm

So if we make the grid cells smaller, we would get better maps, right?

It’s actually not that simple.

The first issue stems from computation load. Increasing resolution drastically increases the amount of memory consumed to track the occupancy grid, and increases computation requirements to keep grid cells updated. The increase in memory consumption is easy to calculate. If we halve the grid granularity from 5cm to 2.5cm, that turns each 5cm square into four 2.5cm squares. Quadrupling the memory requirement for our occupancy grid. Tracking and maintaining this map is a lot more work. In my experience the mapping module has a lot harder time matching LIDAR scan data to the map, causing occasional skips in data processing that ends up reducing map quality.

The second issue stems from sensor precision. An inexpensive LIDAR like my unit salvaged from a Neato robot vacuum isn’t terribly precise, returning noisy distance readings that varies over time even if the robot and the obstacle are both standing still. When the noise exceeds map granularity, the occupancy grid starts getting “fuzzy”. For example, a solid wall might no longer be a single surface, but several nearby surfaces.

High Resolution Fuzzy Map 1cm

As a result of those two factors, arbitrarily increasing the occupancy map resolution can drastically increase the cost without returning a worthwhile improvement. This is a downside to going “too small”, and it was less obvious than the downside of going “too large.” There’s a “just right” point in between that makes the best trade-offs. Finding the right map granularity to match robots and their tasks is going to be an interesting challenge.

(Cross-posted to Hackaday.io)

Phoebe The Cartographer

Once odometry calculation math in the Roboclaw ROS driver was fixed, I could drive Phoebe (my TurtleBot variant) around the house and watch laser and odometry data plotted in RViz. It is exciting to see the data stream starting to resemble that of a real functioning autonomous robot! And just like all real robots… the real world does not match the ideal world. Our specific problem of the day is odometry drift: Phoebe’s wheel encoders are not perfectly accurate. Whether from wheel slippage, small debris on the ground, or whatever else, they cause the reported distance to be slightly different from actual distance traveled. These small errors accumulate over time, so the position calculated from odometry becomes less and less accurate as Phoebe drives.

The solution to odometry drift is to supplement encoder data with other sensors, using additional information can help correct for position drift. In the case of Phoebe and her  TurtleBot 3 inspiration, that comes in courtesy of the scanning LIDAR. If Phoebe can track LIDAR readings over time and build up a map, that information can also be used to locate Phoebe on the map. This class of algorithms is called SLAM for Simultaneous Location and Mapping. And because they’re fundamentally similar robots, it would be straightforward to translate TurtleBot 3’s SLAM demo to my Phoebe.

There are several different SLAM implementations available as ROS modules. I’ll start with Gmapping because that’s what TurtleBot 3 demo used. As input this module needs LIDAR data in the form of ROS topic /scan and also the transform tree published via /tf, where it finds the geometry relationship between odometry (which I just fixed), base, and laser. As output, it will generate an “occupancy grid”, a big table representing a robot’s environment in terms of open space, obstacle, or unknown. And most importantly for our purposes: it will generate a transform mapping map coordinate frame to the odom coordinate frame. This coordinate transform is the correction factor to be applied on top of odometry-calculated position, generated by comparing LIDAR data to the map.

Once all the pieces are in place, Phoebe can start mapping out its environment and also correct for small errors in odometry position as it drifts.

SLAM achievement: Unlocked!

Phoebe GMapping

(Cross-posted to Hackaday.io)

Roboclaw ROS Driver: Odometry Calculation Reversal

rosorg-logo1I’m making good progress on integrating ROS into my TurtleBot variant called Phoebe. I’ve just configured RViz to plot Phoebe’s location as reported by Roboclaw ROS driver‘s odometry information topic /odom. On the same plot, I’ve also placed data reported by Phoebe’s scanning LIDAR. Everything looked good when I commanded Phoebe to drive forward, but the geometry went askew as soon as Phoebe started turning. The LIDAR dots rotated one way, and the odometry reporting arrow rotated another. Something is wrong in the geometry calculation code.

For this particular problem I knew exactly where to start looking. When I assembled Phoebe, I knew I needed to configure motors in the way expected by the Roboclaw ROS driver. The README didn’t explicitly say which motor went on which side, so I went into the source code and got conflicting answers. In the motor driving routine (cmd_vel_callback) it performed calculation for right wheel motion and sent it to motor 1, and left wheel motion went to motor 2.

However, this doesn’t match encoder calculation code. The method EncodeOdom::update_publish expects its parameters to be (enc_left, enc_right). In method Node::run, it retrieves encoder values from motor 1 and saves it in variable enc1, and enc2 for motor 2, then calls update_publish(enc1, enc2). Which would treat encoder value of motor 1 as enc_left instead of enc_right, the opposite of what we’d want.

So the fact data on /odom is going the wrong way was not a surprise, but why didn’t the LIDAR transform also suffer the same problem? This was traced down to an earlier change submitted as a fix for the reverse transform problem – that commit added a negative sign in front of the calculated angle when broadcasting odom -> base_link transform. But that only masked and didn’t fix the underlying problem. The transform might look right, but /odom data is still all wrong, and variables like self.last_enc_right is actually holding the left side value.

The correct fix would be to reverse the parameter order when calling EncodeOdom::update_publish which correctly assigns encoder 1 count the right motor, and encoder 2 the left motor. And with the underlying problem fixed, we no longer need the negative sign so that can be deleted.

With this fix, Phoebe’s laser plot and odometry plot in RViz appear to agree with each other and both correspond correctly to the direction Phoebe is turning. I’ve submitted this fix as a pull request on the Roboclaw ROS driver.

Driving Miss Phoebe (Not Self-Driving… Yet)

Now that the Roboclaw ROS node is configured for Phoebe TurtleBot’s physical layout, and the code is running reliably instead of crashing, I can drive it around as a remote control car. This isn’t the end goal, of course. If all I wanted was a remote control car there were far cheaper and easier ways to get here. Right now the goal is just to verify functionality of the drive system and that it is properly integrated into the ROS infrastructure.

In order to send driving commands to Phoebe, I need something that publishes to the command & velocity topic /cmd_vel. On an autonomous robot, the navigation code would be in charge of sending these commands. But since it’s a pretty common task to move a robot around manually, there are standard ROS packages to drive a robot via human commands. The most basic is teleop_twist_keyboard which allows driving by pressing keys on a keyboard. Alternatively there is teleop_twist_joy for driving via a joystick.

Those remote control (or ‘tele operation’ in ROS parlance) nodes worked right off the bat, which is great. But I quickly got bored with driving Phoebe around on the carpet in front of me. First I launched RViz to visualize scanning LIDAR data as I did before. This was enough for me to drive Phoebe beyond my line of sight, watching surroundings in the form of the laser scan dots. After I verified that still worked, I stepped up the difficulty: I wanted RViz to plot laser data on top of odometry data in order to verify that Roboclaw ROS node is generating odometry data correctly.

To do this, I needed to participate in the ROS coordinate transform stack, the infrastructure to track all the frames of reference for relating robot components to each other in physical space. The Roboclaw ROS node publishes a transform to relate a robot’s position reference frame base_link to the odometry origin reference frame odom. The LIDAR ROS node publishes its data relative to its own neato_laser reference frame. As the robot builder, it was my job to write a transform to relate neato_laser frame to Phoebe’s base_link frame. Fortunately, ROS transform tutorials covered this exact task and I quickly got my desired RViz plot.

RViz with LaserScan and Odometry

It looks like the LIDAR scan plot from earlier, but now there’s an arrow indicating Phoebe’s position and direction. The bigger change not visible in this screen shot is that RViz is now plotting in the odometry frame. This means we no longer watch strictly from Phoebe’s viewpoint  where robot stays in the center of screen. The plot is now in odometry frame, and Phoebe should be moving relative to the map.

I drove Phoebe forward, and was happy to see the laser scan stayed still relative to the map and the red arrow representing Phoebe moved forward. But when I started turning Phoebe, the red arrow turned one way and the LIDAR plot moved the opposite way.

Something is wrong in the coordinate transform data, time for some debugging…

Roboclaw ROS Driver: Add Thread Synchronization

rosorg-logo1I now turn my attention to finding the root cause of random sporadic failures when the Roboclaw ROS driver makes calls into Roboclaw API. The most common symptom of these failures is the TypeError I addressed earlier, but avoiding crash by TypeError is only a band-aid. There were still other issues. Sometimes Phoebe’s movement stutters. And even more mysteriously, sometimes when Phoebe was supposed to be standing still, it would twitch for a fraction of a second.

Looking into ROS logs, the node never intended to send a movement. But it does send a continuous stream of “run at speed X” commands at a rate of ten per second, whether it is trying to move the robot or not. When staying still, that stream continues at the same rate constantly sending “run at speed zero”. The fact that my robot twitches when it’s supposed to be standing still tells me that “run at speed zero” command is occasionally corrupted into a “run at speed X” command. Which starts the motor moving for 1/10th of a second until it is stopped by the next non-corrupted “run at speed zero” command.

Any time there is a random sporadic failure, my instinct is to look for places where threading collisions may be taking place. Programming for multi-threaded environments can get tricky. When I wrote code for SGVHAK Rover, the intent was to make that code easy to understand and pick up. In that spirit, I explicitly kept everything on a single thread to avoid multi-threading issues.

But now we’re playing in the major leagues and there’s no avoiding multiple threads. ROS itself runs across multiple processes and threads, so it could scale to robots that have multiple on board computers, each of which has multiple processing cores. Fortunately, ROS implementation takes care of almost everything, each ROS node just has to make sure they’re doing the right thing with their own private data.

There are at least three different ROS topics involved in this Roboclaw ROS node:

  1. Driving commands received via /cmd_vel
  2. Odometry computation broadcast to /odom
  3. Diagnostics information

Each ROS topic is processed in own thread, so given three topics, we should expect at least three different threads who might all call into Roboclaw API at the same time to perform their tasks. Armed with this knowledge, I looked for code managing cross-thread access. My plan was to review that code to see if I can find any obvious problems with it.

That plan was changed when I found no code managing cross-thread access. I guess its absence qualifies as an obvious problem and it would certainly explain the kind of behavior I saw.

Not being a Python expert, I cruised StackOverflow for a pattern I could use to implement Python thread synchronization. I decided it was most straightforward to use the with keyword described on one of the later replies on this “Semaphores on Python” thread. Using this pattern makes the code change delta very straightforward to read.

There were a few initial calls into Roboclaw API to set things up, I left those alone. But as soon as the code started kicking off events that would have other threads (specifically, when it started the diagnostics thread) every following call into Roboclaw API is synchronized by a threading.Lock() object. With this modification, we can guarantee that only one thread will be performing serial communication to Roboclaw motor controller at any given time, and avoid data corruption by multiple threads trying to talk to the serial port at the same time.

Phoebe ran smoothly and reliably after this work. No more stutter in motion, and no more twitching when standing still. I’ve submitted the fix as a pull request.

Roboclaw ROS Driver: Encoder Count Logging Error

rosorg-logo1Once I fixed the ticks_per_meter error, I could drive Phoebe around by sending movement command messages to the /cmd_vel topic. However, I could not do this for very long (sometimes minutes, sometimes mere seconds) before the Roboclaw ROS driver would crash with the following error:

File "roboclaw_node.py", line 232, in run
    rospy.logdebug(" Encoders %d %d" % (enc1, enc2))
TypeError: %d format: a number is required, not NoneType

Once crashed, Phoebe no longer responds to movement commands, and that is bad. What’s going on here? The instance of TypeError that brought the whole thing crashing down gives us a place to start: one of its %d parameters (which could have been either enc1 or enc2) has a value of None instead of a number representing encoder count.

Looking through the source code file, I could see that the encoder values were initialized to None then replaced with an encoder count by a call to Roboclaw API ReadEncM1 / ReadEncM2. If this call should fail, there’s a try/except block to catch the error, but that leaves the encoder value at None, causing the crash I’ve experienced.

There exists an if statement whose intention might be to avoid this crash by checking for existence of the variable, but I don’t understand how it would help. Encoder count variables would always exist since they were declared and initialized to None.

There are two parts to this problem:

  1. Why is the Roboclaw API call failing?
  2. If it fails, how can we avoid crashing on TypeError?

Part 2 is easier so let’s address that first. TypeError complains when the variables are not numbers, so let’s explicitly check for numbers. Looking through trusty StackOverflow found this method of checking for numeric values in Python, so let’s use that to guard against TypeError.

This change would not address the root cause of API call failing, it’s just a band-aid to avoid fatally crashing and burning. When we fail to retrieve encoder count using Roboclaw API, our internal information become stale which causes more problems down the line. Problems like bad odometry calculations, which will cascade to more problems.

What we really need to do is to address the failing API call, which I’ll work on next. In the meantime this band-aid has been submitted via a pull request.

Roboclaw ROS Driver: Encoder Ticks Per Meter Of Travel

rosorg-logo1Now that Phoebe has taken physical form (imperfect as a first draft tends to be) it’s time to put together software necessary to run it. We’ve already previously established how to use the Neato LIDAR in ROS, so now focus turns to propulsion control via Roboclaw motor controllers. On their product information page, Roboclaw manufacturer linked to this Github repository with the label “Robot Operating System Driver”. So obviously that’s where I should start.

I had to look through this code earlier to learn “Motor 1” of the Roboclaw should be the right hand motor and “Motor 2” the left, but up to this point I’ve only ran this code enough to verify the correct motors turn in the correct direction. Now that I have to configure the driver parameter details to match my physical robot.

A critical parameter is ticks_per_meter, the number of encoder counts that transpire when the robot travels a meter. This encapsulates multiple pieces of information about a particular mechanical implementation: things like motor gear ratio and wheel diameter. This count is required for accurate velocity control, since velocity is specified in meters per second. It is also obviously important to calculate robot odometry, because Roboclaw tracks distance in encoder counts and ROS expects position information in meters.

To test this, I send a command “travel forward at 0.1 meters per second” for one second of time. Configuration is successful when this command makes Phoebe travel 10 centimeters, with corresponding confirmation in output odometry calculations. First I ran the test with default parameters, which resulted in a little over 1 centimeter of travel implying my robot’s ticks_per_meter is almost ten times that of default.

I then re-ran the test with a much higher value for ticks_per_meter parameter and the travel distance didn’t change. Then I tried even larger values, and smaller values, and no matter what the robot kept going forward the same amount. After I double checked that I was spelling my parameter correctly, I went looking into the code.

Thankfully the author had included a line to output parameter to ROS debug log system. (rospy.logdebug("ticks_per_meter %f", self.TICKS_PER_METER)) Once I found this debug message in the log, I was able to confirm that the default value was always used no matter what value I passed in.

Backtracking through the code, I found the cause: The call to retrieve parameter value has a typo, so it is looking for parameter tick_per_meter (singular tick) instead of the intended ticks_per_meter (plural ticks.) Once I added the missing ‘s‘, Phoebe started moving the correct distance. Once verified I submitted this fix via a pull request.

LIDAR Completes First Draft of Phoebe TurtleBot

With the motors connected to Roboclaw, their direction and encoder in sync, and PID values tuned, Phoebe can be driven around via ROS /cmd_vel topic and report its movement via /odom. However, Phoebe has no awareness of its surroundings, which is where the LIDAR module comes in.

Salvaged from a Neato robot vacuum (and bought off eBay), it is the final major component to be installed on Phoebe. Since this is a rough first draft, the most expedient way to install the device is to drill a few holes for M3 standoffs, and mount the module on top of them. This allows the module clear lines of sight all around the robot, while sitting level with the ground. It is also installed as close to the center of the robot as practical. I don’t know if a center location is critical, but intuitively it seems to be a good thing to have. We’ll know more once we start driving it around and see what it does.

By this point the rough draft nature of the project is very visible. The LIDAR spin motor sticks out below the module the furthest, and the motor inadvertently sits right on top of the Raspberry Pi’s Ethernet port, which is the tallest point on a Pi. Raising the LIDAR high enough so they don’t collide left a lot of empty space between the two modules. Which is not wasted at the moment, because the wiring mess is getting out of control and could use all the space it can occupy.

The next version should lay things out differently to make everything neater. In the meantime, it’s time to see if we can drive this robot around and watch its LIDAR plot. And once that basic process has been debugged, that should be everything necessary to enable ROS projects to give Phoebe some level of autonomy.

Phoebe TurtleBot Stage 3 LIDAR

(Cross-posted to Hackaday.io)

Phoebe Receives Raspberry Pi Brain After PID Tuning

Once the motor’s spin direction was sorted out, I connected both encoders to verify A/B signals are in sync with motor direction. Again this is checked by commanding motor movement via Ion Studio software and watching the reported encoder value.

When wired correctly, encoder counter will increase when motor is commanded to spin in the positive direction, and decrease when motor spins negative. If hooked up wrong, the encoder value will decrease when the motor spins positive, and vice versa. The fix is simple: power down the system, and swap the A/B quadrature encoder signal wires.

Once the motor direction is verified correct, and encoder wires verified to match motor direction, we can proceed to the final phase of Roboclaw setup: determine PID coefficients for motor control.

PID tuning is something of a black art. Fortunately, while a perfect tune is very difficult to obtain, it’s not that hard to get to “good enough.” Furthermore, Ion Studio features an “Auto Tune” option to automatically find functional PID coefficients. During SGVHAK Rover construction we had no luck getting it to work and resorted to tuning PID coefficients manually. Fortunately, this time around Ion Studio’s automatic PID tuning works. I’m not sure what changed, but I’m not going to complain.

Once PID coefficients have been written to Roboclaw NVRAM, we no longer need to use the Windows-based Ion Studio software. From here on out, we can use a Raspberry Pi to control our motors. The Pi 3 was mounted so its microSD card remains accessible, as well as its HDMI port and USB ports. This meant trading off access to GPIO pins but we’re not planning to use them just yet so that’s OK.

Software-wise, the Raspberry Pi 3’s microSD card has a full desktop installation of ROS Kinetic on top of Ubuntu Mate 16.04 compiled for Raspberry Pi. In addition to all the Robotis software for TurtleBot 3, it also has a clone of the ROS control node, as well as a clone of the Neato LIDAR control node.

The wiring is not very neat or pretty but, again, this is just a rough first draft.

Phoebe TurtleBot Stage 2 Encoder Pi

(Cross-posted to Hackaday.io)

Establish Motor Directions For Phoebe TurtleBot

The first revision of Phoebe’s body frame has mounting points for the two drive wheels and the caster wheel. There are two larger holes to accommodate drive motor wiring bundle, and four smaller holes to mount a battery tray beneath the frame. Since this is the first rough draft, I didn’t bother spending too much time over thinking further details. We’ll wing it and take notes along the way for the next revision.

Phoebe Frame First Draft.PNG

After the wheels were installed, there was much happiness because the top surface of the frame sat level with the ground, indicating the height compensation (for height difference between motorized wheels and caster in the back) was correct or at least close enough.

Next, two holes were drilled to mechanically mount the Roboclaw motor control module. Once secured, a small battery was connected plus both motor drive power wires. Encoder data wires were not connected, just taped out of the way, as they were not yet needed for the first test: direction of motor rotation.

Phoebe TurtleBot Stage 1 PWM

The Roboclaw ROS node expects the robot’s right side motor to be connected as Motor #1, and the left as Motor #2. It also expects positive direction on both motors to correspond to forward motion.

I verified robot wiring using Ion Studio, the Windows-based utility published by the makers of Roboclaw. I used Ion Studio to command the motors via USB cable to verify the right motor rotates clockwise for positive motion, and the left motor counter-clockwise for positive motion. I got it right on the first try purely by accident, but it wouldn’t have been a big deal if one or both motors spun the wrong way. All I would have had to do is to swap the motor drive power wires to reverse their polarity.

(Cross-posted to Hackaday.io)

Test Frame To Help Arrange Phoebe’s Wheels

Since Phoebe will be a TurtleBot variant built out of stuff already in my parts bin, these parts won’t necessarily fit together well. The first thing to do is to figure out how to make the wheels work together. A simple test frame will mount Phoebe’s two drive wheels and see how they cooperate. And besides, building a two-wheel test chassis is how I’ve started many robot projects and that’s worked out well so far. So now let’s make another one to follow in the grand tradition of two wheel test chassis built to test parts going into SGVHAK Rover and Sawppy Rover.

Phoebe TurtleBot Two Wheel Test Frame

For Phoebe, this simple test chassis established the following:

  • I used a caliper to measure wheel mounting bracket dimensions, and they are accurate enough to proceed. They are spaced the correct distance apart, and their diameter is large enough for M4 bolts to slide through without being so large that the resulting wheel wobbles.
  • The 5mm thick slender connecting members are too weak. The next frame will have greater thickness and generally beefier structure.
  • I wanted a 20cm track. (Left-right distance between wheel centers.) I measured the dimensions for my wheel assembly but the measurements were a little off. Now I know how much to adjust for the next frame.
  • And most importantly: this frame allowed direct comparison of drive wheel resting height against caster wheel height. They were both awkward shapes to measure with a ruler so having the flat surface of a frame makes the measurement easier. Their relative height difference needs to be accounted for in the next frame in order to have a robot body that is level with the ground.

(Cross-posted to Hackaday.io)

Cost to Build Phoebe From Scratch

I chose components for Phoebe (“PB” or Parts Bin) TurtleBot because they were already available to me in one context or another. But not everyone has the same stuff in their own hoard. As an exercise for completeness, below is an estimate of what it would cost to build Phoebe if parts had to be purchased. Naturally, any parts already on hand can be subtracted from the cost. (The expected audience here is likely to have at least a Raspberry Pi 3 and battery to spare.)

Components for parts bin turtlebot phoebe

  • Onboard computer: A Raspberry Pi 3 with microSD card, case, and DC power supply will add up to roughly $50.
  • Laser scanner: LIDAR salvaged off Neato robot vacuum cleaners are on eBay with “Buy It Now” prices of $50-$75 at time of this writing. People living in a major metropolis like Los Angeles can find entire Neato vacuums on Craigslist in a similar price range.
  • Motor controller: A Roboclaw with 7A capacity can be purchased directly from the manufacturer for $70. It is overkill for this project, but it was their entry-level product and it was already on hand. Lower-cost alternatives probably exist.
  • Gearmotor + Encoder + Wheel: Buying the motors I’m using from Pololu would be $35 each without bracket or wheel. However, similar units including mounting bracket and wheel are available on Amazon for $20 each.
  • Caster wheel: A caster wheel can be salvaged off a piece of broken furniture for free. If you have to buy a caster, don’t pay more than $3.
  • Battery: The battery pack I’m using are available for $25 each, but it’s far more battery than necessary for this project. A far smaller pack for $10-15 would be sufficient.

Sum total: $238, which still does not include the following:

  • 3D printer filament.
  • Electrical connectors and wiring.
  • Bolts, nuts, and other assembly hardware.

But given room for improvement (cheaper motor controller and battery) a whole package to build Phoebe from scratch should be possible for under $250, less than half of a TurtleBot 3 Burger.

(Cross-posted to Hackaday.io)

New Project: Phoebe TurtleBot

While my understanding of the open source Robot Operating System is still far from complete, I feel like I’m far enough along to build a robot to run ROS. Doing so would help cement the concepts covered so far and serve as an anchor to help explore new areas in the future.

Sawppy Rover is standing by, and my long-term plan has always been to make Sawppy smarter with ROS. Sawppy’s six-wheeled rocker-bogie suspension system is great for driving over rough terrain but fully exploiting that capability autonomously requires ROS software complexity far beyond what I could handle right now. Sawppy is still the goal for my ROS journey, but I need something simpler as an intermediate step.

A TurtleBot is the official ROS entry-level robot. It is far simpler than Sawppy with just two driven wheels and restricted to traveling over flat floors. I’ve been playing with a simulated TurtleBot 3 and it has been extremely helpful for learning ROS. Robotis will happily sell a complete TurtleBot 3 Burger for $550. This represents a discounted bundle price less than the sum of MSRP for all of its individual components, but $550 is still a nontrivial chunk of change. Looking over its capabilities, though, I’m optimistic I could implement critical TurtleBot functionality using parts already on hand.

Components for parts bin turtlebot phoebe

  • Onboard computer: This one is easy. I have several Raspberry Pi 3 sitting around I could draft into the task, with all necessary accessories like a microSD card, a case, and power supply.
  • Laser scanner: Instead of the Robotis LDS-01, I’ll use a scanning LIDAR salvaged from a Neato robot vacuum that I bought off eBay for experimentation. Somebody has already written a driver package to report its data as a ROS /scan topic and I’ve already verified it sends data that the ROS visualization tool RViz can understand.
  • Motor controller: Robotis OpenCR is a very capable board with lots of expansion capabilities, but I have a RoboClaw left from SGVHAK Rover (JPL Open Source Rover) project so I’ll use that instead. Somebody has already written a driver package to accept commands via ROS topic /cmd_vel and report motion via /odom, though I haven’t tried it yet.
  • Gearmotor + Encoder: TurtleBot 3 Burger’s wheels are driven by Robotis Dynamixel XL430-W250-T serial bus servos via their OpenCR board. Encoders are critical for executing motor commands sent to /cmd_vel correctly and for reporting odometry information via /odom. Fortunately some gearmotor+encoder from Pololu are available. These were formerly SGVHAK Rover’s steering motors, but after one of them broke they were all replaced with beefier motors. I’m going to borrow two of the remaining functioning units for this project.
  • Wheel: Robotis has a wheel designed to bolt onto their servos, and they have mounting hardware for their servos. I’m going to borrow the wheel and mounting hardware from a self-balancing robot toy gathering dust on a shelf. (Similar but not identical to this one.) That toy didn’t have encoders on its motors, but they have the same mounting points and output shaft as the Pololu gearmotor so it was an easy swap. The third wheel, a free wheeling caster, was salvaged from a retired office chair.
  • Chassis hardware: Robotis has designed a modular system (like an Erector Set) for building robot chassis like their TurtleBot variants. As for me… I have a 3D printer and I’m not afraid to use it.
  • Battery: I’ll be borrowing Sawppy’s 5200 mAh battery pack.

This forms the roster for my TurtleBot variant, with an incremental component cost of $0 thanks to the parts bin. “Parts Bin TurtleBot” is a mouthful to say and not a very friendly name. I looked at its acronym “PB-TB” and remembered R2-D2’s nickname “Artoo”. So I’m going to turn “PB” into Phoebe.

I hereby announce the birth of my TurtleBot variant, Phoebe!

(Cross-posted to Hackaday.io)

Examining Basic Requirements For Mapping in ROS

When I first looked at Gazebo robot simulation environment, I didn’t understand how to find the interface layer difference between a simulated robot and a physical robot. Now I’ve learned enough ROS to know: Look at the ROS Node graph to find a node named /gazebo. Anything provided by or consumed by that node is a virtual substitute provided by Gazebo. When the same ROS code stack is on a physical robot, nodes that interface with physical hardware replaces everything provided in simulation by /gazebo.

When I tested a cheap little laptop’s performance in ROS, I put this knowledge to use. Gazebo ran on a high-end computer and every other node ran on the laptop. This simulates the workload of running Gmapping algorithm as if the low-end laptop was mounted on a physical robot. But what, specifically, is required? Let’s look at the ROS node graph once again with rqt_graph. Here’s a graph generated while TurtleBot 3’s mapping demo is running in Gazebo:

TurtleBot 3 GMapping With UI

Here’s a slightly different graph, generated by running the same mapping task but with Gazebo’s GUI and ROS RViz visualization tool turned off. They are useful for the human developer but are not strictly necessary for the robot to run. We see the /gazebo_gui node has dropped out as expected, and the /map topic was also dropped because it was no longer being consumed by RViz for presentation.

TurtleBot 3 GMapping No UI

We can see the Gazebo-specific parts are quite modest in this particular exercise. A physical robot running Gmapping in ROS will need to subscribe to /cmd_vel so it can be told where to go, and provide laser distance scanning data via /scan so Gmapping can tell where it is. Gazebo also publishes the simulated robot’s state via /tf and /joint_states.

And now that I have a spinning LIDAR working in ROS to provide /scan, the next project is to build a robot chassis that can be driven via /cmd_vel. After that is complete, we can use it to learn about /tf and /joint_states.

ROS Is Not News But Shows Up On Hacker News Anyway

ROS is now over a decade old and quite established in its niche. Every day, there are newbies like myself who start their journey to learn the ropes. Motivated by ambition to build software stacks for their own robot projects. Personally, I’ve known about ROS for a long time, though only now am I putting in the effort to work with it firsthand.

When it comes to big software frameworks I definitely count myself as “Grumpy Old Man”. I’ve seen them before, I’ll get proficient at it, and then I’ll move on to the next thing. I don’t get excited… it’s just a tool. But some ROS newcomers get really excited about their discovery! It may not be news in the historical sense, but it’s certainly news to them and they might choose to submit to venues like Reddit or, in today’s example, YCombinator’s Hacker News. And apparently it’s only been about 10 months since the last time someone submitted ROS to Hacker News.

ROS on Hacker News

And since the internet is the internet, the comment thread is naturally filled with positive people espousing the benefits of ROS, thankful of an open source robot infrastructure available so no one has to reinvent the wheel.

Just kidding.

It’s the internet.

The comment thread is actually full of gripes from people critical of the cruft built up during the course of a 10+ year software project. Complaints about how the generalized infrastructure isn’t good enough for their specific needs. Complaints about how it doesn’t use [insert favorite pet technology thing here] without considering its history as of ten years ago. Was it mature? Dependable? Did it even exist back then?

Having worked on big software projects full of legacy code, I look at these “ROS people are idiots” complaints and I shake my head. I know that a software project as big as ROS would have made major decisions based on state of software not ten years ago (when ROS was released) but more like twelve or thirteen years ago when they started putting it together. Plus if they made early decisions based on proven track record, that track record would have stretched even further back.

Yeah, ROS is old. Which is why the Open Source Robotics Foundation, the organization supporting ROS, has focused most of its resources into ROS 2. A ground-up rewrite incorporating the lessons learned during all this time. I’m cautiously optimistic ROS2 will be all it’s promised to be, but we’ll just have to stay tuned to see how reality lines up.

In the meantime, ROS is here, it is mature, and while ROS 2 will be changing a lot of technical fundamentals, it will not change any of the underlying philosophies. So I’ll stay focused learning classic ROS. It has its values, no matter what internet comments say.

Detecting Raspberry Pi Thermal Throttling From Console

Raspberry Pi over temperature 80 85While in the process of obtaining proof that a Raspberry Pi 3 is under-powered for certain ROS processing tasks like mapping, I took a little side trip into the world of Raspberry Pi thermal management. Anyone who has pushed the limits of a Raspberry Pi would have seen a thermometer icon in the upper right corner. A quick search finds that it is put on-screen by firmware and not visible to the operating system. So when a Raspberry Pi is mounted on a robot and not attached to a monitor, we can’t see this icon. However, it is still possible to detect high temperature condition by using the command line tool vcgencmd.

raspberry-pi-logoThis tool appears to be specific to the Raspberry Pi hardware and wraps a collection of tools to query hardware information. Official documentation seems pretty slim, not even an official name. Raspberry Pi forum users hypothesize it stands for “VideoCore General Commands” which is good enough for me.

Raspberry Pi over temperature 85The first useful tool is to measure temperature. vcgencmd measure_temp will return a temperature in Celsius. This internal number is more useful than attaching an external physical temperature measurement because this internal number is what the firmware will use to decide what to do. When temperature rises above 80°C, a thermometer icon overlay with a half-full bar of red is shown on-screen and the system starts pulling itself back. When it hits the target ceiling of 85°C that icon is replaced by one with full bar of red, and the firmware becomes more aggressive throttling things down to stay below 85°C.

It’s possible to keep an eye on temperature by running the tool continuously at a regular interval, something like watch -n 0.5 vcgencmd measure_temp. But once it gets into the 80-80°C range, we can’t tell if the Pi is approaching its limits, or if those limits had been exceeded and the chip slowed itself down to stay in temperature range. For that information, we’ll need a different command.

Running vcgencmd get_throttled will return a hexadecimal number representing a set of binary flags. There is no official documentation of these bits, but a lot of Raspberry Pi user documentation link to this post as reference. If any of the throttling bits are set, the Pi is working at less than maximum potential on account of overheating.

Knowing throttling occurred was enough for my experiment, but if I ever need to go one step further and find out how much throttling has occurred, there are tools for that, too. It’s possible to retrieve CPU clock frequency via vcgencmd get_config arm_freq and also by looking at /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq. But that’s beyond the scope for today.


Symptoms Of A Computer Struggling To Perform ROS Mapping

Once my Dell Inspiron 11 3000 (3180) laptop had its factory installation of Windows safely saved away in a Windows system image backup, its meager 32GB eMMC storage was wiped clean for an installation of Ubuntu 16.04 and ROS Kinetic Kame. Installation was mostly uneventful, except its touchpad stopped working a few minutes after setup began, forcing me to complete setup using keyboard only. This issue seemed to be resolved after updating Ubuntu to latest packages, so it was only a minor annoyance on the way to answering my $130 question: is this meager AMD E2-9000e processor powerful enough to serve as a robot brain? The answer: Yes, but barely.

Dell 3180 SLAM

My test was to run a standard ROS package that performs SLAM (simultaneous location and mapping) using Robotis’ virtual TurtleBot 3 in Gazebo simulation. To reflect the workload of a robot brain running OpenSlam’s GMapping algorithm, I ran only mapping code on the laptop. My desktop computer handled the complex physics simulation and rendering of Gazebo in order to keep the two workloads separate. To give some context for this little laptop’s capabilities, the same mapping workload was run on two other systems for comparison. One faster, and one slower, than this little laptop.

Representing the high end is my desktop computer with an Intel Core i5-7600. It kept up with incoming sensor data effortlessly and matched them up to existing records. Here’s an output log excerpt on the way to generating a high quality map:

Average Scan Matching Score=313.501
neff= 93.949
Registering Scans:Done
update frame 243
update ld=0.0096975 ad=0.30291
Laser Pose= -1.44014 1.54989 -0.977552
m_count 177
Average Scan Matching Score=312.594
neff= 93.949
Registering Scans:Done
update frame 244
update ld=0.0278723 ad=0.896994
Laser Pose= -1.46775 1.55373 -1.87455
m_count 178
Average Scan Matching Score=309.104
neff= 92.9918
Registering Scans:Done
update frame 245
update ld=0.116176 ad=0.441108
Laser Pose= -1.40149 1.64916 -2.31565
m_count 179
Average Scan Matching Score=311.613
neff= 92.9897
Registering Scans:Done
update frame 246
update ld=0.23972 ad=2.51909e-05
Laser Pose= -1.23899 1.8254 -2.31568
m_count 180

There is one interesting observation, though: according to CPU utilization metrics, the ROS Node executing Gmapping consumed 100% of a single CPU core. I don’t know if this means the algorithm has more room for improvement if given a faster core, or if this just means the algorithm takes up as much CPU as it can grab regardless of workload.

On the other end of the performance spectrum is the low power ARM processor of a Raspberry Pi 3, and this SLAM code was too much for the little chip to handle. CPU utilization metrics also show 100% utilization of a single core, but it looks like sensor data avalanches in too quickly for a Pi to process and match up to existing map data. There were only rare successful matches in the sea of errors, as seen in this excerpt of output:

Scan Matching Failed, using odometry. Likelihood=-1430.25
lp:0.367574 2.18056 -1.47424
op:1.16607 1.6985 -3.09185
Scan Matching Failed, using odometry. Likelihood=-1183.3
lp:0.367574 2.18056 -1.47424
op:1.16607 1.6985 -3.09185
Scan Matching Failed, using odometry. Likelihood=-0.874547
lp:0.367574 2.18056 -1.47424
op:1.16607 1.6985 -3.09185
Scan Matching Failed, using odometry. Likelihood=-0.900994
lp:0.367574 2.18056 -1.47424
op:1.16607 1.6985 -3.09185
Average Scan Matching Score=208.949
neff= 55.0344
Registering Scans:Done
update frame 28
update ld=1.28116 ad=2.45138
Laser Pose= 2.08673 0.807569 0.739956
m_count 28
Scan Matching Failed, using odometry. Likelihood=-1376.72
lp:1.16607 1.6985 -3.09185
op:2.08673 0.807569 0.739956
Scan Matching Failed, using odometry. Likelihood=-707.376
lp:1.16607 1.6985 -3.09185
op:2.08673 0.807569 0.739956
Scan Matching Failed, using odometry. Likelihood=-653.378
lp:1.16607 1.6985 -3.09185
op:2.08673 0.807569 0.739956
Scan Matching Failed, using odometry. Likelihood=-116.341
lp:1.16607 1.6985 -3.09185
op:2.08673 0.807569 0.739956

So how did the budget laptop perform in comparison, on its AMD E2-9000e processor?

It was far better than the Raspberry Pi, delivering mostly successful scan matches, but I could see occasional failure messages indicating it was struggling to keep pace with a fire hose of data. Curiously, CPU utilization did not stay pegged to 100%. It sometimes dip as low as 80%, implying there’s another bottleneck in the system keeping the CPU from being fully fed with work. But it’s the results that matter most. A visual examination of the map it generated looks rougher than one generated by my desktop, but usable. Meaning the resulting map might be of “good enough” quality for a robot to use despite its occasional errors.

So the little machine didn’t ace the test, but it managed to squeak by with a passing grade of C+, maybe a B-. This is very encouraging news for performance delivered by a low-end chip. It means we can start experimenting with this inexpensive laptop for now, and we have lots of upgrade headroom in the future.