Recently, I started looking at faster ways to perform dense stereo matching for some work with 3D video. After some experimentation, I found out that by using a selective mode filter paired with naive correspondence matching, I was able to get satisfactory results very quickly. Check out the slide show below for some results!



[red indicates close, blue indicates far away]

 

Here is a download-able Matlab demo, which should work on any pre-aligned stereo image pairs:

stereo_modefilt.zip

The entire code is written in Matlab/C++/MEX. The stereo matching is all in Matlab, and the selective mode filter is coded in C++ and callable from Matlab (meaning it must be compiled before it can run). Currently, the correspondence is the major bottleneck, so anyone who can improve this, please let me know.

This code can be used according to the MIT license. As long as this work is appropriately cited and attributed, and not being used for proprietary or commercial purposes, I’m fully supportive of you using it. Please drop me a line if it helps you!

48 thoughts on “Fast 3D Stereo Vision

  1. I tried d/l’ing from the MATLAB site and the demo wouldn’t run (Undefined command/function ‘edison_wrapper_mex’). Now I downloaded this version and I the demo, again, will not run (Undefined command/function ‘modefilt2_mex’). How do I fix this?

  2. John: I tried to include pre-compiled binaries for the mex file. However, I don’t have binaries for all OS’s. Try:
    >>mex modefilt2_mex.cpp
    Then give it another shot.

  3. Hi,

    I am working on a project on simulating the descent of a lander onto a planet’s `surface, so I was very interested to come across your MATLAB project for Fast 3D Stereo Vision.
    My project is to do with analyzing simulated images of a planetary surface, & calculating a good landing site, for a space craft. We can do this with techniques like shadow mapping & LIDAR.
    I would like to look at Stereo Vision & I was hoping that I could incorporate your MATLAB project for Fast Stereo Vision into my program. Would this be OK with you?

    I have extracted images from my proram & fed them into your project but my early results have been mixed. No fault of your program though! The images that I am using from our planetary surface simulator, are not great. They are quite dark & there is not much contrasting distances – mainly dunes & rocks. The resulting disparity maps have been somewhat unclear, however I am working on it. Any advice/suggestions you might have for alterations to your program, to make it more suitable for planetary surfaces would be greatly appreciated!!

    Thanks again,

    Carmen

  4. You should make sure you set the mins (min pixel disparity) and maxs (max pixel disparity) values correctly for your images, and make sure left and right are passed in correctly.

    Also, try tuning the parameter that controls “window size.” A larger window may give you more accurate results, but a smaller one will give you more detailed results.

    Finally, you may try to pre-process the images to enhance contrast. Good luck!

  5. Hi,

    Some cleaning of my images & tweaking of the window size parameter have improved my results. I just have 1 issue that has been driving me mad. I am trying to save the filtered disparity map to a bitmap file, but the colors are all wrong, as compared to the image displayed in MATLAB. I presume its something to do with colormaps.
    The code I added to your demo.m is:

    colormap(‘default’);
    cmap = colormap;
    imwrite(d, cmap, ‘filtered.bmp’,’bmp’);

    This produces a bitmap but the wide rage of colors that are displayed in the MATLAB image are all shifted to various shades of blue. Have you any ideas?

  6. Try using something like this:

    >>f = getframe(gcf);
    >>imwrite(f.cdata,’filename.png’);

    Using getframe, you should be able to save a file that is EXACTLY what you see on the screen. Good luck!

  7. Hello Shawn,

    I have recently started working on 3D vision using Stereo Disparity.
    I have found you code and your website really useful.

    I will get in touch with you when i need some help.

    many thanks
    naga

  8. Hi Shawn,

    I would like to use your algorithm as a source for computing a depth map-based metric to evaluate the quality of an algorithm I am submitting for publication. You mention that your algorithm is based upon (“Segment-Based Stereo Matching Using Belief Propogation and a Self-Adapting Dissimilarity Measure” by Klaus, Sormann, and Karner). Is there any way that your matlab implementation of this algorithm is cite-able?

    Thank you,
    Kyle

  9. Hi Shawn,

    Although, I can get a depth map finally, but how can I find the largest disparity value(offset value) between Left and Right image?

    Thanks

  10. Hi,i still can not understand how to use displarity plane to fit each segmention region…would you please explain to me? and how to programm it using matlab?
    thank you very much

  11. it is very work , and i need your help,
    f = modefilt2_mex(img,win,ignore);
    this function Undefined and i want what is the meaning by this function and i will try to pre-compiled it.
    please send to me as fast as possible.
    regards
    emy

  12. Hi Shawn! I found ur code very useful for my projects but I’m having problem here I tried ur codes with images from Middlebury database the teddy and cone stereo images. I found put it dosen’t work with your code. I tried changing the window size and set the min and max disparity still fails me.can you help me out? thank you

  13. @Yawn. There is a demo image in the zip file. Otherwise, it may be a trick to get the parameters right for a particular set of images. However, by setting the “maxs” parameter correctly you should be able to get satisfactory results.

    One other note: Make sure your RIGHT and LEFT images are being input correctly.

  14. Hi,shawn lankton

    i still can not understand how to use displarity plane to fit each segmention region…would you please explain to me? and how to programm it using matlab? which part is diparity plane code in your code?

    thank you very much

  15. thank you so much about fast response but , i don’t know how i can compile this file in matlab , when i write this
    >>mex modefilt2_mex.cpp
    the result is
    Error: Could not detect a compiler on local system
    which can compile the specified input file(s) at C:\MATLAB7/bin/win32/mexsetup.pm line 519.
    ??? Error using ==> mex
    Unable to complete successfully

    i think that bacause i type of operating system i used , iam using window xp
    what is your opinion?
    regards
    emy

  16. @ Ray
    thank you so much about your answer, i setup mex and when i write this line
    mex f:\modefilt2_mex.cpp the resulte is
    Error F:\modefilt2_mex.cpp: 82 illegal expression
    Error F:\modefilt2_mex.cpp: 82 found `int’ expected a function
    Error F:\modefilt2_mex.cpp: 82 operands of = have illegal types `int’ and `void’
    Error F:\modefilt2_mex.cpp: 100 illegal expression
    Error F:\modefilt2_mex.cpp: 100 found `int’ expected a function
    Error F:\modefilt2_mex.cpp: 100 operands of = have illegal types `int’ and `void’
    6 errors, 0 warnings

    C:\MATLAB7\BIN\WIN32\MEX.PL: Error: Compile of ‘F:\modefilt2_mex.cpp’ failed.

    ??? Error using ==> mex
    Unable to complete successfully

    so what can i do??!! , i can’t deal with c.
    regards
    emy

  17. Hellow Yawn,
    I have not found the matrix.h, mex.h and except.h files. Please help me where I can find these files to run the programm. I have search these files on google and download, but when I run the programm it gives errors.

  18. Hello Shawn,
    I am a student working on 3D video coding project. Initially I have to get some depth maps. I am using your code to calculate the disparity maps of stereo pairs. I am using the test sequences from http://www.3dtv-research.org/publicSwLibrary.php website. I am not getting good results but I noticed a very strange thing that my results are better when i interchange the input images – Right and left.
    I guess it has something to do with trying out different value for MAXS – I did, but it is not working. Can you advise me on this please? Thank you so much.

  19. I compared the images that you used with my test sequences and I noticed that I need to give the inputs as i did in the image of the second link above. However, the output lacks detail and if I decrease the window size, it does not improve. Also, if you notice the roof of the room in the image(second link), it has blue stripes that follow the texture in the original image. Any ideas how to avoid this?
    Thanks a lot.
    -Karthik.

  20. @Karthik
    Great comments, and thanks for the links to your own examples. You may be right that the ‘left’ and ‘right’ inputs are flipped, but it looks like you figured that out!

    As for improved detail and good performance despite a lack of texture… This simple stereo vision algorithm has it’s limitations! You can refer to the literature (publications like CVPR, PAMI, TIP) to find much more sophisticated algorithms that may help address the problems you mentioned.

  21. Dear Shawn, thank you very much for your rapid response. I guess, this accuracy is enough for my work. However, now that I have the disparity map, I want to calculate the depth map from it. I have all the camera parameters. So how do I calculate the depth maps? I know some formulas that use the disparity value to calculate the depth map,but is there a place where I can find the code to do the same? Or if it is easy to code, can I do it myself? Please advise me. Here are the camera parameters I have:

    f = 24,287 mm (focal length)
    Znear = 0 m
    Zfar = 30m
    Distance between left and right camera: 68mm

  22. Hi, thanks for your code.
    When I tried using the Tsukuba pictures, I get the same result as your picture.
    But if I use your code with my pictures, there are a few problems.
    Matlap give my images the Value 382×512 uint8, in Tsukuba, it is 288x284x3 uint8
    That’s a problem.
    Can you tell me how can I convert my images so that the code is running?

    Thanks for your help.

  23. @congkhanhtruong your images need to be rectified so that they line up on the x axis. You can google to find code that performs this transformation. Also, color images work better because there is more information to use in the registration process. Grayscale images should work too though. (Also, be sure you’re passing the L & R images in correctly… if those are backwards, you’ll get terrible results)

  24. Hi, thanks for your code.

    Can you tell me how can I compile with 64 bits so that the code is running?

    Because when I try to run “demo”, I have this error :

    ??? Undefined function or method ‘edison_wrapper_mex’ for input arguments of type ‘struct’.

    Error in ==> edison_wrapper at 64
    [fimage labels modes regSize grad conf] = edison_wrapper_mex(fim, rgbim, p);

    Error in ==> msseg at 42
    [fimg labels modes regsize grad conf] = edison_wrapper(I,@RGB2Luv, …

    Error in ==> total_stereo at 58
    [segs labels] = msseg(i1,hs,hr,M); %– mean shift segmentation

    Error in ==> demo at 16
    [d p s l] = total_stereo(i1,i2, hs,hr,M,mins, maxs);

    And when I run “compile_edison_wrapper”
    I have this error:

    C:\PROGRA~1\MATLAB\R2009A\BIN\MEX.PL: Error: ‘edison_wrapper_mex.cpp’ not found.

    ??? Error using ==> mex at 218
    Unable to complete successfully.

    Error in ==> compile_edison_wrapper at 5
    mex -O edison_wrapper_mex.cpp …

    I don’t know what I must change here

    mex -O edison_wrapper_mex.cpp …
    segm/ms.cpp segm/msImageProcessor.cpp segm/msSysPrompt.cpp segm/RAList.cpp segm/rlist.cpp …
    edge/BgEdge.cpp edge/BgImage.cpp edge/BgGlobalFc.cpp edge/BgEdgeList.cpp edge/BgEdgeDetect.cpp
    fprintf(1, ‘Done edison_wraper compilation\n’);

    so that the code run with 64 bits !!!!!!!!!!!

    Thanks for your help.

  25. Thanks a lot for your helpful code. I am wondering if it is possible to extract the labelling code from the segmentation code.

    I look forward to hearing from you soon

    thanks for your help.

  26. @Sonia: It sounds like you don’t have MEX properly setup. The main reason why is because when it tries to execute the MEX command, the error outputs indicate that it can’t find MEX, which is only found after you set MEX up. MEX requires that MATLAB knows which C/C++ compiler that you’re using. In the MATLAB command prompt, type in the following (without quotes) “mex -setup”, and go from there. You need to have a C/C++ compiler installed on your computer, or you won’t be able to run the code. Once you do that, run the compile_edison_wrapper again so that the code compiles on your 64-bit machine. It should work after. I’m running Windows 7 64-bit Ultimate, and the code works fine with me.

    @Joseph: The original EDISON code is actually poorly written. It can get a bit obfuscated. However, the labelling is actually not too bad. Once the mean shift segmentation algorithm finishes, be sure to save all of the centroids for each cluster that has been identified. Once you’ve identified all of the centroids, look for points where the joint Euclidean distance (both colour and distance to the centroid) is less than some number (I chose 0.5). If a point’s distance is less than this amount, it belongs to that cluster. Basically, use a loop, have a variable that stores a centroid at a particular iteration, then loop through all points in the image and check to see if points are close to the centroid in question. This brute force technique takes a very long time, so the EDISON authors use efficient nearest neighbour techniques to decrease the amount of time that this executes.

    Hope this helps,
    – Ray.

  27. Hey Shawn,

    May I ask you why there are many wrong small regions at the top right of tsukuba image and can you guess any method to just conceal these errors?
    Thanks

  28. hi
    i have looked ur blog long time…It is a good work…at least i think so…
    rencently i am focusing on stereo matching about Belief Propagation …so do you have touched this method? i have some confused things …
    would you give me an email address…..thx
    best wishes…

  29. hello shawn….
    I am working on project 3D object recognition using stereovision.
    I am having problems in getting the disparity map. can u please tell me how to select the maximum disparity value. Is their some formula to get it.
    And one thing, what should be the maximum distance between the two cameras while making a stereo vision system….
    Please help me…..i am dying to have results….
    thank u….

  30. hi shawn
    Nice work.shawn can u plz guide how can i set maxs and mins accurately according to pick .Any way to find that .i m not getting good resultsss..helpp meeeee outttttt

  31. hi shawn
    Nice work.Shawn plzz Guide how can i set Maxs and mins Accuratelly.Any specific method to calculate that.I m not getting Good Results plzzzzz help me outtt

  32. Hi,
    What paper did you use to do this code, Segment-based stereo matching using beleif propagation and self-adopting dissimilarity measure, or another paper?? If so, can you tell me which, and its link, if possible??

    Thanks for your help

  33. Hi Shawn,

    Great work. I really like the excellent performance of your code.

    However I have two questions concerning the function “stereo.m”:

    1) In line 59
    “pd = shift_image(pixel_dsp,5); %– shift to match i1”
    the disparity map is shifted by 5 pixel. As far as I understood the disparity map should already be aligned according to image i1. So why does one still need to shift the disparity map?

    2) The subfunction “function I = shift_image(I,shift)” (line 98) does not shift the image for a parameter setting of shift=1. In fact it provides the same result as shift=0. Does the subfunction need some modification?

    I hope you have some time to take a look at my questions.

  34. hi? shawn? great work!
    but i have a doubt in the code.

    in the end of function Winner_take_all(), the line

    pd = shift_image(pixel_dsp,5); %– shift to match i1

    why shift the pixle_dsp? why it is 5 pixels?

  35. Hello Shawn, Currently i’m doing masters and my dissertation is on Stereo Vision. I have read your article on Fast Stereo Vision and i found it so useful, Thank you for it.
    I’ve also downloaded your code on it but i can’t run this code in my matlab because i installed windows 8. So i want your help on this problem; What changes should i make to run the program. I’ve also installed MEX compilers but still i face difficulties. Your advise would be so helpful to me.
    Thank You !

Leave a Reply

Your email address will not be published.