How to: Convert an OpenCV cv::Mat to an OpenGL texture

I’m working on using OpenCV to get Kinect sensor data via OpenNI, and needed a way to get a matrix (cv::Mat) into an OpenGL texture – so I wrote a function to do just that – woo! Apologies in advance for the terrible juggling ;-)

The function used to perform the sensor data to texture conversion is:

You can then use the above function like this:

There’s one very important issue to watch out for when using OpenCV and OpenNI together which I’ve commented in the code, but I’ll place here as well as it can be a real deal breaker:

There appears to be a threading issue with the OpenCV grab() function where if you try to grab the device before it’s ready to provide the next frame it takes up to 2 seconds to provide the frame, which it might do for a little while before crashing the XnSensorServer process & then you can’t get any more frames without restarting the application. This results in horrible, stuttery framerates and garbled sensor data.

I’ve found that this can be worked around by playing an mp3 in the background. No, really. I’m guessing the threading of the mp3 player introduces some kind of latency which prevents the grab() function being called too soon. Try it if you don’t believe me!

So just be aware that if you’re using a Kinect you have to be careful with the grab() function… The source code used to create the above video is provided in full after the jump, if you’re interested.

Cheers!

19 thoughts on “How to: Convert an OpenCV cv::Mat to an OpenGL texture”

  1. Thanks, this is exactly what I needed here is a wxGLCanvas derived object that this works in, its not cleaned up but it works great even when resizing etc.

    // Notes:

    // Good info on wxGLCanvas http://wiki.wxwidgets.org/WxGLCanvas#wxGLCanvas_on_a_wxPanel

    // cv::Mat to OpenGL texture http://r3dux.org/2012/01/how-to-convert-an-opencv-cvmat-to-an-opengl-texture/

    // adding glew to the OpenGL SDK http://openglbook.com/setting-up-opengl-glew-and-freeglut-in-visual-c/

    // iplimage Can be converted to an osg::image too http://stackoverflow.com/questions/10876706/converting-iplimage-to-osgimage

    //wxString mystring(glGetString(GL_VERSION).c_str(), wxConvUTF8);
    //char* version = (char*)glGetString(GL_VERSION);
    //wxString mystring = wxString::FromUTF8(version);
    //wxMessageBox(mystring);
    //delete version; 4.3.1 nice

  2. Hi Patrick – good work & glad you found the conversion to texture stuff useful!

    Unfortunately some of the code got “cleaned” on entry, so anything with angle brackets is gone – it’d be great if you could re-post the code and wrap it in pre tags so that it makes it through intact!

    If you put the code as follows it should work (in theory!):

    If it still gets mangled maybe email me the .h and .cpp files (mail-at-r3dux-dot-org) and I can add it from this end.

    Again, thanks for the feedback and example code!

    Cheers!
    -r3dux

  3. Here are the incudes in “…” style:

    // DetectGLCanvas.h

    #pragma once

    #include “wx/glcanvas.h”
    #include “wx/dcclient.h”
    //
    //
    //// OpenGL view data
    //struct GLData // todo get rid of this into the class itself
    //{
    // bool initialized; // has OpenGL been initialized?
    // float beginx, beginy; // position of mouse
    // // float quat[4]; // orientation of object //todo remove this entire struct
    // // float zoom; // field of view in degrees
    //};

    namespace cv{ // Forward declaration of the Mat object within the cv namespace
    class Mat; //…

    // —————————–

    // DetectGLCanvas.cpp

    #pragma once

    #include “glew.h” // This needs to be included before GL/gl.h and that is included in App.h along with wx/glcanvas.h
    #include “detect_gl_canvas.h”
    #include “App.h”
    #include “GL/glu.h” //this is included in glew so it doesnt need to be here soon

    #include “opencv2/core/core.hpp”
    #include “opencv2/core/mat.hpp”
    #include “opencv2/highgui/highgui.hpp”
    #include “opencv2/imgproc/imgproc.hpp”

    #include “opencv2/contrib/contrib.hpp”
    #include “opencv2/objdetect/objdetect.hpp”
    #include “opencv2/features2d/features2d.hpp”
    //#include “opencv2/core/gpumat.hpp”
    //#include “opencv2/gpu/gpu.hpp”

    #include “wx/app.h”
    #include “wx/msgdlg.h” //todo remove later

    BEGIN_EVENT_TABLE(DetectGLCanvas, wxGLCanvas)
    EVT_SIZE(DetectGLCanvas::OnSize) //…

  4. Hi,

    I dont have Kinect,
    I try use my webcam and i have a white screen (no texture), only the OpenGL square.
    I just comment the 2 lines in main function:

    //capture.retrieve(camFrame, CV_CAP_OPENNI_BGR_IMAGE);
    //capture.retrieve(depthFrame, CV_CAP_OPENNI_DISPARITY_MAP);

    And change this in main function:

    cv::VideoCapture capture;
    capture.open(0);

    capture >> camFrame;

    // Draw texture contents
    draw(camFrame, camFrame);

    Any sugestion?

  5. Hi!
    I’m trying to implement on my PC with nvidea GeForce 560 your function “matToTexture”, but I’ve error – glGenerateMipmap: identifier not found.
    What can I have a problem?
    Thank you.

    1. If I remember correctly the glGenerateMipmap function isn’t in the core set of OpenGL commands, so you’ll need to enable an extension to use it.

      The easiest way to do this is to include GLEW in your project, and then initialise GLEW early on in your code – which will automatically make extensions available as needed. Using GLEW is simple, and there’s some example code you could look at here: Simple Texture Loading with DevIL Revisited, or of course there’s always Google =D

      Alternatively, you could just not use mipmaps by setting the minification and/or magnification filters to use something like GL_LINEAR instead of GL_LINEAR_MIPMAP_LINEAR or such, at which point you can completely remove the call to glGenerateMipmap!

      Hope this helps.

      1. Thanks for you last answer, but now I have a new problem with glGenerateMipmap:
        simpleGL.obj : error LNK2001: Unresolved external symbol __imp____glewGenerateMipmap
        I included all libraries.
        May be you have any idea?

  6. hi!
    I am trying to do it from OpenGL –> OpenCV
    I am recovering an image from RealSense with OpenGL and I want to use it and work with it in OpenCV. any hint?
    Thanks!

Leave a Reply

Your email address will not be published.