Fast Level Sets Demo

The links below point to the technical report and a demo written in C++/MEX that can be run directly in MATLAB. The demo implements the Chan-Vese segmentation energy, but many energies can be minimized using the provided framework.

Sparse Field Method – Technical Report [pdf]
Sparse Field Method – Matlab Demo [zip]

To run the MATLAB demo, simply unzip the file and run:
>>sfm_chanvese_demo
at the command line. On the first run, this will compile the MEX code on your machine and then run the demo. If the MEX compile fails, please check your MEX setup. The demo is for a 2D image, but the codes work for 3D images as well.

My hope is that other researchers wishing to quickly implement Whitaker’s method can use this information to easily understand the intricacies of the algorithm which, in my opinion, were not presented clearly in Whitaker’s original paper. Personally, these codes have SUBSTANTIALLY sped up my segmentations, and are allowing me to make much faster progress towards completing my PhD!

Thanks to Ernst Schwartz and Andy for helping to find small bugs in the codes and documentation. (they’re fixed now!)

For more information regarding active contour, segmentation, and computer vision, check here: Computer Vision Posts

The two functions that I couldn’t find, and missed the most (especially when writing hack-y code for class projects) were median filtering and morphological dilation. So, in hopes of sparing other the pain of writing them… here they are! The function medfilt_dilate.py has both functions.

medfilt_dilate.py

The medfilt() function uses the PIL filtering code. The dilate() function was written from scratch with NumPy.

Here is a download-able Matlab demo, which should work on any pre-aligned stereo image pairs:

stereo_modefilt.zip

The entire code is written in Matlab/C++/MEX. The stereo matching is all in Matlab, and the selective mode filter is coded in C++ and callable from Matlab (meaning it must be compiled before it can run). Currently, the correspondence is the major bottleneck, so anyone who can improve this, please let me know! Enjoy.

This idea arose when I was trying to de-noise some images as well as do some in-painting of “bad” pixels (that have no value). Consider the image below. The darkest-blue areas are bad pixels. We have no information for those pixels. The other pixels are colored to show how far that pixel is from the camera. (see the post on Stereo Vision) However, some of the good pixels still have the wrong value. These are the noise pixels.

[Initial Image]

In the rest this post I talk about how we use a selective mode filter to convert the above image into the one below. (There’s also download-able Matlab/C++ code)

[Final Result of Selective Mode Filter]

First Try

First, we compare a standard median filter along-side a standard mode filter. We see that the two results are very similar. I deem the mode filter to be slightly better, but this is pretty subjective. As you can see, both methods do a good job of eliminating noise although neither filter removes the bad pixels.

  
[Standard Median Filter, Standard Mode Filter]

Being Selective

The solution is to ignore bad pixels so that they can not contribute to the computation of the median or mode (depending on which filter you use). In my example, bad pixels are value ‘0′ while all the other pixels are positive and non-zero. Therefore, I re-wrote the mode filter to ignore pixels under a threshold when computing the local mode around each pixel. The result is very nice. Some of the fine-scale features are lost, but overall, the number of noise pixels and bad pixels are greatly reduced.

Conclusion & Download

This is a simple idea, but it had very nice results. A good next step would be to create a selective median filter. I implemented this in mex so the operation is very quick. In the demo below I include binaries for most platforms, but if doesn’t work on your computer you may need to re-compile.

modefilt.zip

UPDATE:
My new post: Sparse Field Active Contours
implements quicker, more accurate active contours.

Today, I added demo code for the Hybrid Segmentation project. This segmentation algorithm (in the publications section) can be used to find the boundary of objects in images. This approach uses localized statistics and sometimes gets better results than classic methods. For an example, see the video below: The contour begins as a rectangle, but deforms over time so that it finally forms the outline of the monkey.

This can be used to segment many different classes of image. To try it out, download the demo below and run >>localized_seg_demo

localized_seg.zip

This code is based on a standard level set segmentation; it just optimizes a different energy. I’ve also made a demo which implements the well-known Chan-Vese segmentation algorithm. This technique is similar to the one above, but it looks at global statistics. This makes it more robust to initialization, but it also means that more constraints are placed on the image. Download it and see what you think! Again, unzip the file and run >>region_seg_demo

sfm_chanvese_demo.zip (New! Described Here)

regionbased_seg.zip (old and slow)

For another Matlab implementation of Active Contours check out: James Malcolm’s Webpage. He has some codes for very fast approximate implementations as well as a full numerical implementation.

What This Post Teaches

In this post, I show how to create a mex file, how to set up inputs and outputs, how to get access to Matlab objects, and how to manipulate them. I also give a skeleton mex program that might be helpful. There is a lot more to learn, and I’d refer you to the mex manual regardless.

What You Should Already Know

I’m going to assume you already know how to program in C, and already know how to program in Matlab. This post does not teach either language, but will show you how to use them together to get the best of both worlds.

Hello world!

Lets get started. First, type:
>>mex -setup
Then go through the menus to select a compiler that you have installed on your system. I use gcc on linux, visual studios on windows, and mexopts on mac. Now, create a file called, “helloworld.cpp” and we’ll start coding. In this first example, pay attention to the signature of the mexFunction. This signature is always the same.


#include <math.h>
#include <matrix.h>
#include <mex.h>

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
  mexPrintf("Hello World!\n");
}


Save the file, and in the Matlab prompt type,
>>mex helloworld.cpp
If all goes well, you should get the following when you run helloworld:
>>helloworld
>>Hello World!

Setting Up Inputs and Outputs

Now that we have a working program, the next step is to get inputs and outputs going so we can do something useful. The mxArray pointers plhs and prhs represent a pointer to the left hand sside and a pointer to the right hand sside respectively. The left hand side are the outputs and the right hand side are the inputs.

put something like this in the top of mexFunction:


  //declare variables
  mxArray *a_in_m, *b_in_m, *c_out_m, *d_out_m;
  const mwSize *dims;
  double *a, *b, *c, *d;
  int dimx, dimy, numdims;
  
  //associate inputs
  a_in_m = mxDuplicateArray(prhs[0]);
  b_in_m = mxDuplicateArray(prhs[1]);
  
  //figure out dimensions
  dims = mxGetDimensions(prhs[0]);
  numdims = mxGetNumberOfDimensions(prhs[0]);
  dimy = (int)dims[0]; dimx = (int)dims[1];
  
  //associate outputs
  c_out_m = plhs[0] = mxCreateDoubleMatrix(dimy,dimx,mxREAL);
  d_out_m = plhs[1] = mxCreateDoubleMatrix(dimy,dimx,mxREAL);


Accessing and Manipulating Matlab Objects

To access the variables, you need to associate a pointer to the data in the mxArray. Once you do this, accessing the variables is very simple.


  a = mxGetPr(a_in_m);
  b = mxGetPr(b_in_m);
  c = mxGetPr(c_out_m);
  d = mxGetPr(d_out_m);


Now it is possible to access the arrays with standard C or C++ [] notation. There are three important things to remember though:

• You use 0-based indexing as always in C
• You still use column-first indexing like in Matlab, though
• To access the arrays, you use linear indexing (you can’t use [x][y], you have to use [y+x*dimy]

With those three things in mind, go crazy. You can use standard C libraries (as long as you include them). You can use for loops as much as your heart desires, and your code will be much, much faster than its Matlab equivalent.

Downloads

Download this starter file (demo.cpp). This has all of the features discussed in this post, and should be a good primer for any mex function. Give it a try. Tell me what you think, and let me know if any improvements are needed.

demo.cpp

To run the demo, download the .cpp file, then at the prompt run:

  >>mex demo.cpp
  >>a = round(rand(2)*10)
  >>b = round(rand(2)*10)
  >>[c,d] = demo(a,b)

GrowCut Region Growing Algorithm

This algorithm is presented as an alternative to graph-cuts. The operation is very simple, and can be thought of with a biological metaphor: Imagine each image pixel is a “cell” of a certain type. These cells can be foreground, background, undefined, or others. As the algorithm proceeds, these cells compete to dominate the image domain. The ability of the cells to spread is related to the image pixel intensity.

The authors give some pseudocode that very concisely describes the algorithm.


//for every cell p
for all p in image
  //copy previous state
  labels_new = labels;
  strength_new = strength;
  // all neighbors q of p attack
  for all q neighbors
    if(attack_force*strength(q)>strength_new(p))
      labels_new(p) = labels(q)
      strength(p) = strength_new(q)
    end if
  end for
end for


Segmentation Results

Once implemented, this is a nice way to get segmentations. It is quite fast, and the initialization is very intuitive. Consider this picture of a lotus flower:

I made an initialization by clicking 20 points in the flower and 30 points outside. I then made a “label map” where unlabeled pixels are 0 (gray), foreground pixels are 1 (white) and background pixels are -1 (black).

Based on this simple initialization, we obtain a very decent segmentation:

As you can see, it isn’t perfect, but it is quite good. Its possible to interactively refine the seed points to improve the segmentation, but I didn’t do that here.

Matlab Code Downloads

I implemented this code in Matlab (using mex files due to the extensive use of for loops). You can download this below with compiled binaries for mac, linux, and windows. Unzip the file and run >>growcut_test for a demo.

UPDATE: I’ve fixed some bugs thanks to reader, Lin. The code works much better now!

Source & Compiled Binaries (96k) [zip]
“GrowCut” Paper [pdf]

Please let me know if you find this useful, and if you make improvements! Also, check out these related segmentation posts:

Related Segmentation Posts

• Mean-Shift Image Segmentation
• Localized Active Contours
• Fast Level Sets for Segmentation
]]> http://www.shawnlankton.com/2008/03/growcut-segmentation-in-matlab/feed/ 14 http://www.shawnlankton.com/2008/02/quick-and-simple-derivatives-in-matlab/ http://www.shawnlankton.com/2008/02/quick-and-simple-derivatives-in-matlab/#comments Fri, 08 Feb 2008 06:10:30 +0000 malcolm http://www.shawnlankton.com/2008/02/quick-and-simple-derivatives-in-matlab/ Ever try to compute the directional derivatives on a matrix? Google turn up menacing formulas for Taylor expansions, grid spacing, and boundary conditions?

A quick and simple way of computing derivatives is to perform arithmetic on shifted versions of the matrix, and vectorized indexing help Matlab speed things up. For example, use the following for the (central) x-derivative:

dx = (M(:,[2:end end]) - M(:,[1 1:end-1]))/2

You can even define inline functions to perform the shift and derivative operations. Below are definitions for the shift operations and first and second order derivatives.

% shift operations
shiftD = @(M) M([1 1:end-1],:);
shiftL = @(M) M(:,[2:end end]);
shiftR = @(M) M(:,[1 1:end-1]);
shiftU = @(M) M([2:end end],:);

% derivatives
Dx = @(M) (shiftL(M) - shiftR(M))/2;
Dy = @(M) (shiftU(M) - shiftD(M))/2;
Dxx = @(M) (shiftL(M) - 2*M + shiftR(M));
Dyy = @(M) (shiftU(M) - 2*M + shiftD(M));
Dxy = @(M) (shiftU(M) - shiftD(M) + shiftL(M) - shiftR(M))/4;

And use them like this:
>> Ax = Dx(A)
Ax =
-7.0000 -6.5000 5.5000 5.0000
3.0000 2.5000 -1.5000 -1.0000
-1.0000 -1.5000 2.5000 3.0000
5.0000 5.5000 -6.5000 -7.0000

>> Ayy = Dyy(A)
Ayy =
-11 9 7 -5
15 -13 -11 9
-9 11 13 -15
5 -7 -9 11


]]> http://www.shawnlankton.com/2008/02/quick-and-simple-derivatives-in-matlab/feed/ 1 http://www.shawnlankton.com/2007/12/3d-vision-with-stereo-disparity/ http://www.shawnlankton.com/2007/12/3d-vision-with-stereo-disparity/#comments Wed, 19 Dec 2007 22:00:16 +0000 Shawn Lankton http://www.shawnlankton.com/2007/12/3d-vision-with-stereo-disparity/ 2D is nice, but these days I’m getting interested in doing computer vision in 3D. One way to get 3D data is to use two cameras and determine distance by looking at the differences in the two pictures (just like eyes!). In this project I show some initial results and codes for computing disparity from stereo images.

Introduction

UPDATE: Check this recent post for a newer, faster version of this code. The new version no longer relies on mean-shift.

People can see depth because they look at the same scene at two slightly different angles (one from each eye). Our brains then figure out how close things are by determining how far apart they are in the two images from our eyes. The idea here is to do the same thing with a computer. Check this for some information on the geometry and mathematics of stereo vision. First, here are the images I’ll use to show results.



These two images are slightly different. The top one is from the left and the bottom is from the right. It’s a bit hard to see the disparity like this, so here are the same two images placed “on top” of one another.



You can see that the close-up objects like the lamp are very misaligned in the two images, while the farther-away things like the poster and the camera are lined up better. The greater the misalignment, the closer the object.

This pair of images is one of many standard stereo pairs that can be found at the Middlebury stereo vision site. These guys keep a compendium of standard datasets as well as a scoreboard of who’s algorithms work the best. The algorithm I talk about here is a knock-off of the one that was on top in December 2007: “Segment-Based Stereo Matching Using Belief Propogation and a Self-Adapting Dissimilarity Measure[PDF]” by Klaus, Sormann, and Karner. (Mind that the algorithm here is *inspired* by the algorithm of Klaus et al. Theirs is much more complete)

Getting Pixel Disparity

The first step here is to get an estimate of the disparity at each pixel in the image. A reference image is chosen (in this case, the right image), and the other image slides across it. As the two images ’slide’ over one another we subtract their intensity values. Additionally, we subtract gradient information from the images (spatial derivatives). Combining these gives better accuracy, especially on surfaces with texture. In the video below, we can see a visualization of this process. You’ll notice how far-away objects go dark (meaning they line up in the two images) at different times than close-up objects. We record the offset when the difference is the smallest as well as the value of the difference.

We perform this slide-and-subtract operation from right-to-left (R-L) and left-to-right (L-R). Then we try to eliminate bad pixels in two ways. First, we use the disparity from the R-L pass or the L-R pass depending on which has the lowest matching difference. Next, we mark as bad all points where the R-L disparity is significantly different from the L-R disparity. Finally, we are left with a pixel disparity map.



In this image, red-er colors indicate closer pixels, and blue-er colors represent pixels that are farther away.

Filtering the Pixel Disparity

In the next step, we combine image information with the pixel disparities to get a cleaner disparity map. First, we segment the reference image using a technique called “Mean Shift Segmentation.” This is a clustering algorithm that “over-segments” the image. The result is a very ‘blocky’ version of the original image.

Then, for each segment, we look at the associated pixel disparities. In my simple implementation, we assign each segment to have the median disparity of all the pixels within that segment. This gives the final result:



Here again, the red colors are close objects, and blue objects are far away.

Matlab Code

I spent some time getting these simple ideas into working form. I’ve posted the codes and images I used as well as a demo script. To see how the code works, simply un-archive everything and run demo from a Matlab prompt. Enjoy, and let me know how these work for you.

stereo_modefilt.zip (New! Described here.)

lankton_stereo.tar.gz (Old)

Check out the newest results:



[red indicates close, blue indicates far away]

Conclusion

This stereo algorithm is just a tool to be used on other projects. For instance, by computing the stereo disparity of a stereoscopic video it is possible to improve tracking results by using the 3D information. Also, segmentations can be made more accurate if 3D information is known.

I wanted to put this up to introduce people to stereo vision (as this was my introductory project). Hopefully, the words and the codes above will save you some time getting up to speed. I would ask that if you find this helpful and make improvements that you let me know!

]]> http://www.shawnlankton.com/2007/12/3d-vision-with-stereo-disparity/feed/ 34 http://www.shawnlankton.com/2007/11/mean-shift-segmentation-in-matlab/ http://www.shawnlankton.com/2007/11/mean-shift-segmentation-in-matlab/#comments Fri, 16 Nov 2007 21:43:00 +0000 Shawn Lankton http://www.shawnlankton.com/2007/11/mean-shift-segmentation-in-matlab/ Background

Recently I have decided to explore tracking from 3D point clouds extracted from stereo vision cameras. Step 1: Extract 3D point cloud from stereo vision cameras. So right now I’m implementing Segment-Based Stereo Matching Using Belief Propogation and Self-Adapting Dissimilarity Measure” by Klaus, Sormann, and Karner. This paper is defined by the source on stereo vision to be the best one around. This paper has two parts. Part 1: Segment the image. Part 2: Compute disparity (and depth) from the segments. Well, today I finished Part 1.



First Try

The authors refer to a mean-shift segmentation algorithm presented in Mean Shift: A Robust Approach Toward Feature Space Analysis” [pdf] by Comaniciu and Meer to do the image segmentation. This paper (unlike some of my own previous work) leans towards oversegmentation of an image. Meaning that you prefer to get lots of little bits rather than the “right object” after the algorithm has run.

Well, after looking over the paper and getting a grasp for the mathematics, I took a crack at implementing it. Easily done… HOWEVER, my first attempt, written in Matlab, was painfully slow. (For a simple image it took 6 hours to run!) So, I got on the internet and came up with a better solution!

The Solution

Some great guys at Rutgers University implemented this paper in C++ and made the code available to the public under the name EDISON. (there’s also a nice GUI that goes along with this if you want to just play to see if these codes will work for you). Okay, so I had C++ codes that worked well (only 2 sec to do an image rather than 6 hours). The next step was to bring the code into Matlab.


These were the type of results I was trying for

I cracked my knuckles and got ready to write a MEX wrapper for this EDISON code. Then I said to myself, “Self, maybe you should check the ‘net first.” Turns out I had a good point. I found the website of Shai Bagon. Mr. Bagon had already made the MEX wrapper! Awesome.

I downloaded the codes and put them together. Mr. Bagon’s stuff worked right out of the box, although it would have saved me about an hour if I would have had this information (alternative readme.txt for Matlab Interface for EDISON). I also wrote my own wrapper-wrapper so that I could process grayscale images, and do simpler calls to accomplish what I wanted. If you’d like the code, download my wrapper-wrapper here (msseg.m).

Results

Here is a sample of the output of this algorithm. The first image is a regular photo of some posed objects. The second image is the segmented version. Notice how the regions of the image are much, much more constant. This image has been broken into “tiles” of constant color.



The original image (part of a standard pair of test images).

The segmented image (ready to be processed in step 2)

Conclusion

Don’t re-invent the wheel. Taking a first crack at the implementation was good, and it helped me understand the algorithm. However, there was no need for me to spend a week tweaking it to be super-fast or two days getting the Matlab interface working. These things had already been done! It feels nice to knock out a task that you thought was going to take a week in a few hours : ) Stay tuned for the stereo part of this paper coming soon. Then maybe people will be writing about my page!

]]> http://www.shawnlankton.com/2007/11/mean-shift-segmentation-in-matlab/feed/ 24