Augmented Reality Built On “Structure From Motion”

When learning about a new piece of technology in a domain I don’t know much about, I like to do a little background research to understand the fundamentals. This is not just for idle curiosity: understanding theoretical constraints could save a lot of grief down the line if that knowledge spares me from trying to do something that looked reasonable at the time but is actually fundamentally impossible. (Don’t laugh, this has happened more than once.)

For the current generation of augmented reality technology that can run on cell phones and tablets, the fundamental area of research is “Structure from Motion“. Motion is right in the name, and that key component explains how a depth map can be calculated from just a 2D camera image. A cell phone does not have a distance sensor like Kinect’s infrared projector/camera combination, but it does have motion sensors. Phones and tablets started out with only a crude low resolution accelerometer for detecting orientation, but that’s no longer the case thanks to rapid advancements in mobile electronics. Recent devices have high resolution, high speed sensors that integrate accelerometer, gyroscope, and compass across X, Y, and Z axis. These 9-DOF sensors (3 types of data * 3 axis = 9 Degrees of Freedom) allow the phone to accurately detect motion. And given motion data, an algorithm can correlate movement against camera video feed to extract parallax motion. That then feeds into code which builds a digital representation of the structure of the phone’s physical surroundings.

Their method of operation would also explain how such technology could not replace a Kinect sensor, which is designed to sit on the fireplace mantle and watch game players jump around in the living room. Because the Kinect sensor bar does not move, there is no motion from which to calculate structure making SfM useless for such tasks. This educational side quest has thus accomplished the “understand what’s fundamentally impossible” item I mentioned earlier.

But mounted on a mobile robot moving around in its environment? That should have no fundamental incompatibilities with SfM, and might be applicable.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s