Introduction

UPDATE: Check this recent post for a newer, faster version of this code. The new version no longer relies on mean-shift.

People can see depth because they look at the same scene at two slightly different angles (one from each eye). Our brains then figure out how close things are by determining how far apart they are in the two images from our eyes. The idea here is to do the same thing with a computer. Check this for some information on the geometry and mathematics of stereo vision. First, here are the images I’ll use to show results.

These two images are slightly different. The top one is from the left and the bottom is from the right. It’s a bit hard to see the disparity like this, so here are the same two images placed “on top” of one another.

You can see that the close-up objects like the lamp are very misaligned in the two images, while the farther-away things like the poster and the camera are lined up better. The greater the misalignment, the closer the object.

This pair of images is one of many standard stereo pairs that can be found at the Middlebury stereo vision site. These guys keep a compendium of standard datasets as well as a scoreboard of who’s algorithms work the best. The algorithm I talk about here is a knock-off of the one that was on top in December 2007: “Segment-Based Stereo Matching Using Belief Propogation and a Self-Adapting Dissimilarity Measure[PDF]” by Klaus, Sormann, and Karner. (Mind that the algorithm here is *inspired* by the algorithm of Klaus et al. Theirs is much more complete)

Getting Pixel Disparity

The first step here is to get an estimate of the disparity at each pixel in the image. A reference image is chosen (in this case, the right image), and the other image slides across it. As the two images ‘slide’ over one another we subtract their intensity values. Additionally, we subtract gradient information from the images (spatial derivatives). Combining these gives better accuracy, especially on surfaces with texture. In the video below, we can see a visualization of this process. You’ll notice how far-away objects go dark (meaning they line up in the two images) at different times than close-up objects. We record the offset when the difference is the smallest as well as the value of the difference.

We perform this slide-and-subtract operation from right-to-left (R-L) and left-to-right (L-R). Then we try to eliminate bad pixels in two ways. First, we use the disparity from the R-L pass or the L-R pass depending on which has the lowest matching difference. Next, we mark as bad all points where the R-L disparity is significantly different from the L-R disparity. Finally, we are left with a pixel disparity map.

In this image, red-er colors indicate closer pixels, and blue-er colors represent pixels that are farther away.

Filtering the Pixel Disparity

In the next step, we combine image information with the pixel disparities to get a cleaner disparity map. First, we segment the reference image using a technique called “Mean Shift Segmentation.” This is a clustering algorithm that “over-segments” the image. The result is a very ‘blocky’ version of the original image.

Then, for each segment, we look at the associated pixel disparities. In my simple implementation, we assign each segment to have the median disparity of all the pixels within that segment. This gives the final result:

Here again, the red colors are close objects, and blue objects are far away.

Matlab Code

I spent some time getting these simple ideas into working form. I’ve posted the codes and images I used as well as a demo script. To see how the code works, simply un-archive everything and run demo from a Matlab prompt. Enjoy, and let me know how these work for you.

stereo_modefilt.zip (New! Described here.)

lankton_stereo.tar.gz (Old)

Check out the newest results:

Conclusion

This stereo algorithm is just a tool to be used on other projects. For instance, by computing the stereo disparity of a stereoscopic video it is possible to improve tracking results by using the 3D information. Also, segmentations can be made more accurate if 3D information is known.

I wanted to put this up to introduce people to stereo vision (as this was my introductory project). Hopefully, the words and the codes above will save you some time getting up to speed. I would ask that if you find this helpful and make improvements that you let me know!

Introduction

There is a movie theater in Atlanta that is nearly impossible to park at. It requires you to drive around endlessly to find a space. It occurred to me one evening, that with cameras set up to watch the lot it would be possible to track all the cars and know where the available spaces were. Then, movie-goers could get the map of all available spaces via an MMS message.

When I took “Technology Ventures” last semester in the College of Management a team and I worked together to design a business plan around this technology called “Park-Find”. Over the course of the project we researched our competition, researched our target market, made analyses of costs to develop and produce the technology, and learned a great deal about what goes into planning a business.

Here I’ll summarize some parts of the business plan. The full document, as well as a final presentation of the business (similar to a presentation that could be given to VC’s etc.) are available for download. Finally we decided that although the technology could make money we didn’t want to launch the company because it seemed a bit too risky to out-weight the potential rewards.

Abstract

The pay parking management equipment industry of the United States is a $3.2 billion industry with immense potential for additional revenue. Current systems exist to provide total parking system management, but they fail to provide an efficient, secure system that is easy to use for both parker and lot owner. Park-Find offers a solution that will change the way people think about parking. We offer a unique technology that allows parking lot owners to actively monitor and manage their lots via an autonomously running video tracking system. With Park-Find, customers will increase their profitability with a solution that provides more benefit to parkers. Our industry analysis indicates that there is significant need for this type of product, and that there exists an opportunity to penetrate the market with our novel solution. Because of our strong intellectual property and strategic positioning in the market, we expect Park-Find to grow and become profitable as well as becoming an attractive acquisition to established parking solution providers. Through careful positioning, we expect liquidation in the next five years.

In the business plan we address, among other things, the following areas:

• Executive Summary
• Business Description
• Product Offerings
• Market and Industry Analysis
• Industry Background
• Target Market
• Market Size
• Market Trends
• Competitor Analysis
• Product Appeal
• Marketing Plan
• Strategic Partnerships
• Pricing
• Promotion
• Placement
• Sustaining Competitive Advantage
• Operations
• Outsourcing and Strategic Partnership
• Research and Development
• Business Milestones
• Risk Assessment
• Financial Plan
• Exit Strategy

Below are selected sections from the business plan.

Venture Summary

Parking is something that everyone is familiar with and nearly all of us do every day. We have been trained to use various methods of payment and control that are scattered around to help owners and managers of parking systems deal with the volume of parkers that use their facilities everyday. Over the years, the technology used to manage parking has been changing. Spring-loaded mechanical parking meters led to digital parking meters. Next, ticket-based machines were introduced, which can sometimes take cash or even credit card thus allowing someone to park without carrying a pocket full of change.

Park-Find is prepared to enter the market with the next step in parking management solutions. Using our innovative technology, the gates and tickets of today will vanish. No longer will people be required to wait in long lines to enter and exit. Parking places can be found automatically. Parking as an industry becomes more profitable, and parking your car becomes less of a chore.

This dream is made possible by recent developments in the field of computer vision. New algorithms allow computers to take video from small wireless cameras placed in a parking facility, process the information, understand it, and produce a simple map of which spaces are available and which spaces are occupied. From here, this map can be integrated with existing payment systems to streamline the process of payment, simplify management and enforcement, as well as give parkers hints about where to park to save time and effort.

Value Proposition

The parking management equipment industry is very large ($3.2 billion annually), and is perpetually re-invested in by pay-parking facilities nationwide. Right now the market is in upheaval as companies are searching for a technology solution that fits the needs of the industry. Park-Find, using its innovative vision-based solution, has the opportunity to leap past other technology providers in the industry and give customers a solution that far out-performs the status quo and even other radical technologies.

Based on industry research and engineering experience we expect the Park-Find product offering to have the ability to save our customers considerable amounts of money and thereby increase their profits. We can do this for a lower cost and with greater profit margins than most competitors in the field. These factors blend to create a valuable opportunity for Park-Find to enter this market, help customers solve their problems, and in the process grow into a valuable company.

Core Technology::Park-Watch Engine

The Park-Watch Engine is the heart of Park-Find’s product offering. This system will use visual tracking algorithms to monitor cars in the parking structure. Then based on their movements, a virtual map that shows the location of occupied and unoccupied spaces can be continuously updated. The information contained in this map can then be used in various ways to help manage parking at the facility.

Downloads

• Park-Find Business Plan (892k, pdf)
• Park-Find Business Plan, Financial Appendicies (148k, pdf)
• Park-Find Presentation Slides (892k, pdf)

Introduction

So, I’ve said before that Computer Vision is teaching computers to understand images.  Well, I’m currently working on a slightly new problem.  Now I’m trying to teach computers to understand movies!

The idea in most visual tracking is: given the location of an object in the first frame of the movie, follow it through the whole video.

Applications

This has lots of applications.  Many of them are defense-related, but tracking is also very important in medicine, security, robotics, manufacturing, exploration, etc. etc…

How it works

Yikes! I’m pretty busy learning about that right now, but I plan to put some tutorials and source code here soon.  In the meantime, feel free to email me.

Results

Ahhh, here’s the good part!  Videos!  What you see here are black and white videos of either cars or people.  The colored boxes represent where the computer has determined the "targets" to be.  You can see that the boxes follow the "targets" as they move around in the scene.

Conclusion

I hope you liked these results… There will be lots more to come soon, as this is my new primary research focus.

Introduction

One of the main aspects of computer vision is finding the appropriate outline of shapes in images. This is useful for analyzing “natural images” like the ones taken from a camera or medical images such as 3D MRI scans of the brain. This project began by trying to define the boundary of a particularly tricky structure in the brain called the putamen.

I first developed a solution that worked, then analyzed it to figure out the mathematical principles behind what I was doing. In the end, a very nice technique was discovered that is both highly effective and mathematically sound. This idea was first submitted to a conference in August 2006. It was subsequently published in March 2007.

Shawn Lankton, Delphine Nain, Anthony Yezzi, Allen Tannenbaum. Hybrid Geodesic Region-Based Curve Evolutions for Image Segmentation. In Proceedings of SPIE Medical Imaging, San Diego, 2007. [pdf]

(Poster presented at conference.)

Below is a very high-level summary of the paper along with some results and a code demo.

Abstract

This paper presents a novel segmentation technique (Segmentation is the process of detecting the boundary of an object in an image. This can be 2D or 3D images). This technique is based off of two well known classes of techniques, but combines the two in a clever way that allows it to capture benefits of both methods. These two methods are: gradient-based active contours, and region-based active contours. The mechanism of our solution is to start from a naive initial curve, and then move each point on the curve based on analysis of the statistics of the local interior and exterior regions.

Level Set Active Contours

Active contour methods begin with an initial curve and define some cost for that curve based on its geometric properties and the associated image data. Cost based on the geometry is provided to keep the curve smooth, and cost based on the image data are intended to attract the contour to object boundaries. This curve is then deformed in the way needed to decrease the cost thus moving the curve toward a position where cost is locally minimized. Presumably this occurs when the curve is correctly situated on the object.

Related Work

Essentially what we are doing is combining the ideas of “Edge-based” flows and “Region-based” flows.

Edge-based flows use an edge detector to find edges, then try to fit a curve nicely on those detected edges. The problem is that these methods only look at very local image information. That means that it is very susceptible to noise, and the initial placement of the curve. For a reference, check this paper: Geodesic Active Contours by Caselles, et al.

Region-based flows take an alternate approach. They try to model the global characteristics of entire regions of the image. This approach is nice because its very robust to initialization and noise, but sometimes images can’t be modeled well in terms of global rules and erroneous segmentations occur. For a reference, check this paper: Active Contours Without Edges by Chan and Vese

The Hybrid Flow

The hybrid flow aims to blend the benefits of the geodesic active contours and the region based active contours. This is accomplished by forming a cost based on a shortest weighted path. The weights at each point along the path are determined from local regions around the curve. The resulting flow is more robust to initial curve placement and image noise like region-based flows, but also capable of finding significant local minima and partitioning the image without making global assumptions about its makeup.

The key assumption that we make about ob jects to be segmented by this technique is this: At each point on the true edge of an object, nearby points inside and outside the ob ject will be modeled well by the mean intensities of the local regions. The result is an energy that is more global in nature than edge-based flows.

The cost of the entire curve is often called an ‘energy.’ In the definition of this hybrid energy we use several notations. I represents the image, and x and s represent independent spatial variables. Omega and Omega-bar represent the interior and exterior of the curve respectively. Chi, in the equations below, represents a ball centered at the point s that specifies the local region being examined.

We define the ul and vl functions:

in terms of local sums (SIl and SEl) and local areas (AIl, AEl). Based on these notations, we can define the hybrid energy:

In this energy we use an outer integral over s to cover the points along the curve. Then, in the inner integrals we use x to inspect the entire domain restricted by the Chi function so that the only contribution in from image information in the local neighborhood around the point s. The energy around each point s is a function of how well the interior and exterior are modeled by their means, ul and vl. The total energy is the sum of the contribution of energy from every point around the curve.

Using the calculus of variations, it is possible to take the derivative of this energy with respect to the shape of the curve itself. Hence, we find the way to deform the curve such that this energy is always decreasing. By decreasing the energy in an iterative fashion, we continuously refine our guess at the correct answer until we finally converge on the local minimum solution.

Results

Here are two experiments run with this flow. In each set, you see the initial contour (in red), and then the result of three different techniques: a typical region-based technique, an edge-based technique, and the presented hybrid technique. In all cases, the hybrid method outperforms all others.

First a synthetic image designed specifically to foil the other algorithms.

The next experiment is an MRI image of the putamen. This is a small structure in the sub-cortex of the brain.

As you can see, there is significant improvement with the hybrid method.

Conclusion

The presented technique combines the ideas of geodesic and region-based active contours and as a result produces a segmentation algorithm which has benefits of both. Like other geodesic models, this approach is capable of looking locally for correct solutions while only making weak assumptions about global image properties. Also, as with region based models, our method has increased robustness to noise and reduced dependence on initial curve placement as a result of taking image data from local regions. The algorithm has proven to be more versatile than either of the two standard techniques presented, even in its simplest form.

Despite its benefits, this method still has some drawbacks. As with all geometric-based energies, initial curve placement is still important. Although this algorithm is less dependent than some, it is still necessary to initialize the contour nearby the object to be segmented or risk that the final segmentation result will converge at an incorrect local minima. Additionally, the key assumption made about image makeup is not ideal for all images. There are cases where the ideal border of an ob ject is not characterized by a separation of image intensities at that border.

The method presented here is merely the first application of a new class of energy functionals based around combining local and global flows. The ability to define a geodesic energy with respect to local regions of image data has many possible applications. Extension of the implementation to accommodate higher order statistics, vector valued images, and images of higher dimensionality will improve performance and open the door to other potential applications.

Many of these improvements will be addressed in my future publications.

Demo

Check out this Matlab code to see an example of this technique in action.

localized_seg.zip

Feel free to contact me with any questions, comments, or improvements on the code!

Introduction

This was a class project for a grad class at Georgia Tech on robotics. The course was a bit of a survey, and as such, the final project is a bit reflective of this. It covers a little bit of everything from vision (my specialty), to tracking (my other specialty… now), to locomotion, to neural networks. It was a ton of fun to do… enjoy!

Below is a description of the project along with our solutions to several problems… Finally, at the end you can find our project code.

I’d like to send a shout out to my two esteemed collegues on this project: Brad Schwagler, and Brian Stefanovic.

Now, before we start, I’d like to whet your appetites with a little video demo:

Motivation

This robot, is really just a cute project to play around with the Aibo. However, we made up a whole “back story” so, here it is: There are disorganized objects in the world such as litter, household clutter, etc.

It sure would be nice if there was a robot that could find, classify, acquire and put away that stuff! Lets make the Aibo do that! So, we used green geometric shapes to represent “trash.” The dog finds, classifies, and picks up the trash, then brings it to a base station.

Locomotion & Implementation

We used URBI & liburbi for Matlab for all the programming. I’m a big fan of Matlab, of course, and the URBI package made interfacing to the Aibo a breeze. URBI also has a bunch of built-in functions for locomotion. Also, you can use the Webots simulator to play with the Aibo virtually… This is great if A) you don’t have an Aibo, or B) you don’t want to deal with charging its batteries!

Sensing

We used rangefinders for a basic collision avoidance and for localizing on the object when we were ready to pick it up. To do this, we recorded the distance the robot sees when looking “at infinity” as its head sweeps around Then, we compare that with the values we get in situ to make sure there’s an object there.

We also use some cooler sensors like stereo microphones, and vision.

Audiolocation

We wanted the robot to be able to return to a “home base” with his collected “trash.” To do this, we had the base broadcast a tone ( we wanted it to be supersonic, but the mics weren’t good enough, so we used 2kHz).

Then, based on the phase difference in the two microphones, we determined the angle to the source. This is a technique known as Phase Interferometry. Below are some phase plots from the left and right ears… As you can see, there’s not a huge amount of difference to work with.

Vision

Since we cheated a little and made all of our objects green, segmentation was simplified somewhat. Here’s a sample image from Aibo’s camera:

We mapped our image to a chromatic color space known as YBR. This has three components like the familiar RGB, but Y is intensity (the grayscale version of this image), and B and R are the percentage of red and blue respectively that make up the intensity. Here are the three channels we get:

Notice how the box stands out a bit better in this space (its the only green thing). Now, based on some training data, we produced a probability distribution function for the objects very similarly to these guys. Based on this probability we can extract something like this:

Then, with a little morphology, we turn it into this beauty:

These segmentations are important for tracking the object (we extract its centroid), and for classification… You’ll see that one in a bit..

Tracking

So, with these segmentations the tracking is pretty simple. We use this handy little algorithm:

“Find Object” step consist of taking an image, segmenting it, and finding how many degrees from center the object is. Then if we loose the object we look around for it. This ends up being pretty tedious because the Aibo locomotion package we were using wasn’t great.

Classification

The geometric shapes that represented different kinds of “trash” were cubes, spheres, and tetrahedrons. We used a neural net for classifying the objects. We tried a bunch of features before finding one that worked well. The trick finally ended up being border angles. We computed the angle from one point to the next along the border of the shapes. The results look a bit like this:

Notice the four levels for the cube, the sinusoid for the sphere, and the three levels for the tetrahedron. This ended up being a great way to do shape classification with neural networks!

We used a backpropogation network with 70 inputs and two outputs (00 = cube, 01 = sphere, 10 = tetra). Our network had 150 nodes in the hidden layer. We did the whole implementation with the Matlab toolbox.

Also, to make this work, we needed a lot of examples, both positive and negative. We also added noise to our training set. In the end we were getting 95+% success. (That was with a best-of-five-classifications scheme that took 5 different segmentations).

Conclusion

All in all, the project was a total success. We accomplished everything we set out to do and got an A! Also, we learned a lot and entertained our classmates with a live demo in class.

It would be fun to try to do this with fewer assumptions (more complex vision, more object classes, etc.) Also, the Aibo wasn’t really the best platform for this application. A wheeled robot would have been much more capable (but far less cute).

I was impressed with how well the neural network worked. It was really amazing the results we were getting by the end.

Files

As usual, here are some nice files to help you out… This code is *terribly* commented, and probably not readable. However, here it is if you’re looking for guidance:

project.zip (374 Kb)

Introduction

I’ve spent plenty of time working on robotics projects, and I have enjoyed all aspects of the design process from mechanical to electrical to computer science. However, if you are only interested in the computer science part (control theory, artificial intelligence, creating behavior) then having to have an electromechanical platform to run your code on can be a big hassle!

One solution to this is to use a pre-made platform (such as in this project). However, this too can be problematic because everything can break, and when developing algorithms, it is often convenient to prototype them in a simulated environment where real-life confounders don’t hinder your creativity.

Simulations

This problem is often solved by robot simulators. One that I’ve used with great success is differential steering model. This is a simple, but very useful model for robotic actuation. With this type of drive system the robot can drive straight, trace an arc, or even turn in place.

Simulated World

The simulated robot can make its way around a simulated environment represented by a .png file where black represents walls, and white represents flat, clear terrain. Thus any “environment” can be shown as a black and white image. For testing, I used two types of terrains, an organic-looking obstacle course and an “office” with square walls and corners.

Navigation

So, now that we have a simulated robot in a simulated world we want it to do something. The goal I was working towards was this: Make the robot wander around the environment covering most of the area and not hitting any of the walls. This is accomplished with a pretty simple algorithm:

1. Go straight unless sensors are blocked
2. If one sensor is blocked, turn away from it
3. If both are blocked, turn towards the farthest
4. Turn until both sensors are free
5. Continue going straight

There is a lot of room for improvement here (obviously), but this was a very quick project.

Results

Check out a couple of videos. Here we see the robot’s path as it drives around the area.

Click To Play

Click To Play

Demo

Download the demo and run “>>setup.m” Please feel free to make improvements to this. If you do, let me know!

robot_simulator.zip

Introduction

What started as a class project has developed into a prototype algorithm for an actual medical instrument. The motivation is simple. Elderly people, wheelchair bound individuals, amputees, and many other people suffer from large wounds known commonly as pressure ulcers or bed sores. These wounds (apart from being very nasty) can be quite painful and take a very long time to heal. Because the healing process is so long, the wound must be measured daily to determine if interventions used by the medical personnel are helping to shrink it.

Usually wound area and volume are measured. Currently, when area is measured, it is often done by measuring length and width and modeling the wound as a rectangle. I created a computer algorithm that is capable extracting the area with very small error (less than 5%).

Preliminary Solution

I used a simple algorithm that takes examines the image and makes a guess about the wound outline. This initial version still has some problems, but the end results were very encouraging.

Conclusion

The end result was a total success! In addition to performing quite well, this project impressed my professor so much that I was offered a graduate research assistantship position in the lab where I now work.

This project is now being maintained and even drastically improved by grad students working at CATEA and CCG Group at Georgia Tech.

Introduction

This originated as a final project for a microcontrollers class I was in. Consequently, this project is designed on a PIC microcontroller. The electronics are built onto a small board that plugs into the computer’s PS2 port and provides another PS2 port for the keyboard to plug in to. While plugged in it secretly steals passwords. I’ll give some background about PS2 keyboard protocol as well as info about electrical and software design. Finally, I’ll provide links to schematics and source code.

I want to give credit to Ryan Westafer as well. He was my partner on this project.

PS2 keyboard protocol

The keyboard uses two signals to implement its synchronous communication protocol. Both clock and data lines are open-collector and therefore idle high. When either party is ready to send data, the clock is driven low for eight bits of data plus a start bit, stop bit, and parity bit. This protocol is used to send ‘make’ and ‘break’ codes for each keystroke. Make codes signify a key has been pressed, and break codes signify its release. Here is a list of all make and break codes. In fact, this guy’s site explains all of this very well. Link.

Electronic Design

I wanted to use the 5V power that is supplied in the PS2 connector to power our device so it could be covertly implanted. I also didn’t want to mess-up the computer or keyboard, hinder the user, or give ourselves away. This didn’t really require sneakiness because I just hooked clock and data lines to our high impedance digital inputs on the PIC. I also added an LED for debugging purposes, but it would definitely be removed in a for-real spy version. Schematic.

Software Design

The algorithm works by watching for clock edges then recording the eight data bits that follow. The program then checks to see if the received byte is within the range of acceptable characters (0×15-0x7d). If the byte is within the given range of alphanumeric characters, it gets processed. If not, the byte gets ignored. Ignored bytes are either break codes, or function key make codes, and thus are of no interest. Once bytes are received and validated, the pattern recognition algorithm determines if data logging should begin.

Pattern recognition algorithm is a big word for a simple state machine that looks for patterns like “root”, “su”, or other programmable phrases that signify that a pattern is coming. When a pattern is recognized, the next 16 characters are stored to a RAM buffer.

The guidelines of the class were that two PIC processors had to communicate, so here ends the functionality of the ‘spy board.’ The spy board could be connected to another board (which had a LCD display) via an SPI connection. When a button on the spy board was pushed it downloaded its ram buffer to the larger board. The spy board just kept the make codes for the keys. Once they were transmitted to the larger board, a lookup table was used to determine which make code made what ASCII character. This message was then displayed on the LCD display.

Stuff I learned

Turns out that mouse movement drives the same clock line as the keyboard. Either they share the clock, or the OS requests the state of keys such as “ALT” or “CTRL” when the mouse is moved. Because the data line remains high during this occurrence, null bytes are received. Fortunately the algorithm, designed to discard invalid bytes, ignores the mouse movement.

Also, after a byte is received you need to wait approximately 30 ms in order successfully ignore all non-make code transmissions of the keyboard.

Furthermore, I learned during this project that holding a chip onto a socket != putting the chip in that socket. *Especially* when trying to program a chip. It ended up causing all sorts of errors and once I started actually plugging it in development went much faster.

Future Work and Other Ideas

Ideally, I would like to re-build this and bundle it with a flash memory card so that it could just record every thing that the snooped on computer typed. Then, when a ‘secret code’ was typed in it would dump everything back onto the PS2 line and print out a complete log. Then you could get passwords and all sorts of other stuff.

Also, if I could do it all over again, I would use C instead of assembly. (C makes life immensely easier!)

Code & Schematic Files

PIC16F676 Code
PIC18F452 Code
Schematic

Introduction

As part of the IEEE hardware team I’ve done a lot of projects, but I’m proudest of this one. This was a PIC based system that implemented a closed-loop PID controller for position and speed control of two DC motors. In addition, the control loop could be adjusted to induce arcs.

The system was designed as a peripheral for another processor so it also had an SPI command interface that allowed it to be communicated with on a high level while it controlled the motors on a low level. The peripheral had a serial console that allowed you to tune, test, and drive the robot.

Shout outs go to Dr. John Peatman and my esteemed partner in this endeavor Jacques Fortier.

Motivation

In order to complete its task, the robot needed to have precise control of its speed and position. Because much of the way finding of this robot was accomplished through dead reckoning, it was also especially important to maintain accurate information regarding the distance traveled by the robot.

In order to give the robot all of these features, we designed a special sub-processor that was devoted to the low-level control of the two primary motors. We implemented this system on a Microchip PIC18F252. This processor was housed on the same board as the motor driver which took control signals from this controller and used them to actually drive the motors.

Our sub-processor monitored the motors, and applied an appropriate pulse width modulated (PWM) signals to allow them to reach their goal. This controller also had the ability to communicate with the primary controller to receive speed and distance commands as well as report accomplished speed and distance traveled back to the commanding processor.

Feedback

In order to get speed and distance information from the motors, we purchased motors with built in encoders that output a quatrature encoder interface (QEI) signal. This signal is comprised of two pulse trains that tell when the wheel travels a part of a rotation, and the direction of motion. When channel A makes a low to high transition, the value of channel B is checked to determine the direction. A sample QEI output can be seen below.

The motion controller was able to interrupt on low to high transitions from the QEI on channel A and create an estimate of instantaneous speed and direction by monitoring the number of counts obtained during a specified time interval. We used this information as inputs to the control loop that determines and compensates for errors in speed or distance.

The Control Loop

We implemented a PID (Proportional, Integral, and Differential) control loop to control the speed of the motors. The speed of the motors is manipulated by altering the duty cycle of a PWM signal generated by the processor. The duty cycle of this signal is known as the control value. below is a functional diagram of the control loop.

This algorithm first determines the difference between the commanded speed and the actual speed. This value is known as error. The integral error is determined by adding the error to an accumulator, and the derivative error is calculated by subtracting the error from the previous error.

Every time the control loop executes, each of these errors are multiplied by a gain constant that has been tuned to optimize response time and accuracy. These errors, multiplied by their gain constants are then be added to the control value to compute the new control value.

In this way, the control value is continuously updated based on the response of the motors. This ensured that the motors are moving at the desired speed despite drag, obstacles, or other unexpected track conditions.

A final compensation factor is summed along with the P, I, and D errors. This is an error generated by inconsistencies in the two wheels. This helps to ensure that the robot goes straight by compensating for factors that slow one wheel but not the other.

Position Control

This was accomplished by controlling the speed based on the robot’s proximity to the desired position. The motor controller could take commands from the main processor that told it how many ‘ticks’ to travel. This value corresponded exactly to the number of QEI pulses that the robot should move.

The exact distance represented by each tick was determined by the resolution of the encoders and the diameter of the wheel. Our control system was capable of responding to 2944 ticks per revolution. With wheels that were 2.2 inches in diameter, the total distance per tick was 2.346×10-3 inches. Thus, very detailed motor control was possible.

The position control algorithm allowed the robot to approach its target distance at any speed. Then, as the target approached, it would slow its speed so that it could stop accurately when desired. This was achieved with a simple algorithm diagramed below.

The motors position was monitored and continuously updated in local variables. Once the motors reached a distance a certain threshold before the desired position, the position control section of the algorithm would command the motors to stop. This would ensure that as the motors slowed down and achieved a speed of zero the robot would be at the desired position. These two systems were independent for the left and right wheel so that they could be commanded to go different distances for turning purposes.

Communication

These abilities provided the robot with the tools needed to control the speed and distance of travel. However, in order to be truly useful, this sub processor needed to be able to receive commands from the master processor, and provide information regarding the motor status back.

We accomplished this by using an SPI (Serial Peripheral Interface) communication protocol. The communication was message-based. The main processor could send eight-byte messages to the PID controller. These messages began with a one byte op-code telling the sub processor what type of command the message contained. The next seven bytes held data such as the desired speed, or the desired distance to be traveled.

The sub processor could simultaneously send back eight bytes that held instantaneous speed, and total distance traveled information. With the ability to control speed, position, and communicate with the main processor completed, the motion control system was almost done.

Tuning the Control

The final step was to tune the PID control algorithm. This is accomplished by adjusting the gain constants by which each error is multiplied by before summing it with the previous control effort. This is a complicated process that is mostly achieved through trial and error.

Another tool that helped was using GNUplot to make graphs of debug data output by the PID controller. This allowed us to see the interaction of different errors. This interaction can be seen below.

The work of tuning the processor was simplified by developing a user interface on the PID controller that was separate from the main processor. This allowed the control system to be tuned over serial by adjusting gain constants stored in EEPROM.

With the motor controller fully tuned, it performed very well and provided excellent speed and position control for the robot without bothering the main processor with this low level task. Full code for this motion controller is below.

The Motor Driver

All of this fluffy code stuff is great, but you always need some good ol’ power electronics too. The motor driver was required to step up logic level PWM signals to 9.7V motor driving PWM at up to 3A. In order to accomplish this, we used an L298 H-bridge chip and some glue logic. In addition, it was a nice place to house the PIC chip that implemented the PID controller. We added some handy plugs for programming, connecting to other processors, and attaching motors and power. Just because I’m a nice guy, I’ve included a schematic and layout for the board as well as the code below.

Downloads

PID C Code (8kb)
Motorboard Schematic (184kb)
Motorboard Layout (54kb)

Final Thoughts

I was really happy with this project. Below you can see me and Jacques giving our ‘gang sign.’ In case you can’t read it that says “PIC” … as in the PIC chip we wrote the PID controller on.

But more seriously, there were some valuable lessons learned. I really got the hang of SPI. It may have been better to use I2C for this, but I’ll have to save that for next time. I also got really turned on to systems and controls. The idea of a closed loop control system that uses its error to correct itself is very clever and elegant. I strongly recommend taking a class on systems and controls if you haven’t yet.

Another thing I’d like to share is *ALWAYS* program in C. Assembly is sooo much more annoying. We started the project in assembly and quickly realized that things were getting out of hand so we switched to C (God’s own language) and everything started going very smoothly.

If I had it to do over again, I might also make the position control better. As it was, you really needed to tweak constants every time you changed the default speed of the system. It was highly accurate, but it would be a little more robust if I had implemented a PID position control algorithm in addition to the PID speed control algorithm.

Introduction

The purpose of signals is to communicate information. Some of this information is designed for humans to receive. Pictures and music are some examples. Most others are intended for computers or machines to receive. In this exercise the ability to create and decode touch tones, a task that happens millions of times a day, was replicated using principles of filtering.

First touch tones had to be understood and synthesized, and then using filtering each had to be decoded to produce the original key sequence that generated it. The end result was a dialing and decoding system like that of a phone.

Generating Touch Tones

A touch tone is made up of two frequencies corresponding to the keys’ position on the grid of the phone. For instance, 5 is produce by playing a pure 1336 Hz sine wave and a pure 770 Hz sine wave simultaneously. The result is the warbly sound you hear.

dtmfdial.m

Making Filters

The first step to decoding the touch tones was to design a set of filters whose purpose it would be to eliminate all frequencies but one. To accomplish this, a bandpass filter was used. Eight filters were constructed to correspond to the eight possible frequencies that the touch tones are made up of. Each filter had to be scaled so that the magnitude of its frequency response was equal to exactly one. The final code was as follows:

dtmfdesign.m

The frequency response of each filter is important because it shows what every possible frequency component will be scaled by. For instance, a signal with a frequency that occurs where a filter is peaking will go through virtually unmolested because it will be scaled by a factor of one. On the other hand, signals with frequencies that occur where the frequency response is very low will be almost eliminated by the filter because they will be scaled by a small number. The images below show the passbands of all 8 filters that are used.

These filters had to be narrow enough that they would not overlap. It was imperative that each filter would only select one of the 8 frequencies that make up DTMF. Below is an image of the filters that finally accomplished that task.

Making Sense of The Signal

The second phase of the decoding algorithm involved a scoring function. This function took in one impulse response of a bandpass filter and one short segment of a touch tone and determined if there was a frequency in the pass band after the two were convolved.In order to determine if a frequency was in the pass band the magnitude of the convolution was measured. If it was above 0.71 then the frequency was considered to be passed otherwise it wasn’t. The final code was as follows:

dtmfscore.m

Putting it Together

Finally, a main function was written to call the other functions. It parsed the input signal of touch tones using a program called dtmfcut, and then used loops to iterate each tone through each filter. The resulting scores from those iterations were used to decode the row and column of each key, and thus make a vector containing the original keys that were dialed. The completed program was tested by running it on a signal of synthesized touch tones. The touch tones were created with the string ‘891#40789132*DABC’ which produced a vector whose spectrogram is shown below.

dtmfrun.m
dtmfcut.m

Final Thought

Millions of people use phones many times a day and yet, very few understand what complex processes go on inside the touch tone creation and decoding process. In this exercise, that understanding at least has been obtained. However, the ideas presented have a far broader application. The ability to use FIR filters to encode and decode signals in order to accomplish goals is pretty darn important, and it was fun to do too!

For some other great projects in Matlab, check these out:

3D Vision with Stereo Cameras
Simple Robot Simulator
Neural-Net Powered Aibo Robot Project