r3dux.org

A number-pimping side project from the valleys in *NEW* upside-down flavour.

  • Home
  • ABOUT
  • OLD SITE
  • SEARCH
  • FEEDBACK

How to: Convert an OpenCV cv::Mat to an OpenGL texture

r3dux | January 14, 2012

I’m working on using OpenCV to get Kinect sensor data via OpenNI, and needed a way to get a matrix (cv::Mat) into an OpenGL texture – so I wrote a function to do just that – woo! Apologies in advance for the terrible juggling ;-)

YouTube Preview Image

The function used to perform the sensor data to texture conversion is:

// Function turn a cv::Mat into a texture, and return the texture ID as a GLuint for use
GLuint matToTexture(cv::Mat &mat, GLenum minFilter, GLenum magFilter, GLenum wrapFilter)
{
	// Generate a number for our textureID's unique handle
	GLuint textureID;
	glGenTextures(1, &textureID);
 
	// Bind to our texture handle
	glBindTexture(GL_TEXTURE_2D, textureID);
 
	// Catch silly-mistake texture interpolation method for magnification
	if (magFilter == GL_LINEAR_MIPMAP_LINEAR  ||
	    magFilter == GL_LINEAR_MIPMAP_NEAREST ||
	    magFilter == GL_NEAREST_MIPMAP_LINEAR ||
	    magFilter == GL_NEAREST_MIPMAP_NEAREST)
	{
		cout < < "You can't use MIPMAPs for magnification - setting filter to GL_LINEAR" << endl;
		magFilter = GL_LINEAR;
	}
 
	// Set texture interpolation methods for minification and magnification
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, minFilter);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, magFilter);
 
	// Set texture clamping method
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, wrapFilter);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, wrapFilter);
 
	// Set incoming texture format to:
	// GL_BGR       for CV_CAP_OPENNI_BGR_IMAGE,
	// GL_LUMINANCE for CV_CAP_OPENNI_DISPARITY_MAP,
	// Work out other mappings as required ( there's a list in comments in main() )
	GLenum inputColourFormat = GL_BGR;
	if (mat.channels() == 1)
	{
		inputColourFormat = GL_LUMINANCE;
	}
 
	// Create the texture
	glTexImage2D(GL_TEXTURE_2D,     // Type of texture
	             0,                 // Pyramid level (for mip-mapping) - 0 is the top level
	             GL_RGB,            // Internal colour format to convert to
	             mat.cols,          // Image width  i.e. 640 for Kinect in standard mode
	             mat.rows,          // Image height i.e. 480 for Kinect in standard mode
	             0,                 // Border width in pixels (can either be 1 or 0)
	             inputColourFormat, // Input image format (i.e. GL_RGB, GL_RGBA, GL_BGR etc.)
	             GL_UNSIGNED_BYTE,  // Image data type
	             mat.ptr());        // The actual image data itself
 
	// If we're using mipmaps then generate them. Note: This requires OpenGL 3.0 or higher
	if (minFilter == GL_LINEAR_MIPMAP_LINEAR  ||
	    minFilter == GL_LINEAR_MIPMAP_NEAREST ||
	    minFilter == GL_NEAREST_MIPMAP_LINEAR ||
	    minFilter == GL_NEAREST_MIPMAP_NEAREST)
	{
		glGenerateMipmap(GL_TEXTURE_2D);
	}
 
	return textureID;
}

You can then use the above function like this:

// Create our capture object
cv::VideoCapture capture( CV_CAP_OPENNI );
 
// Check that we have actually opened a connection to the sensor
if( !capture.isOpened() )
{
	cout < < "Cannot open capture object." << endl;
	exit(-1);
}
 
// Create our cv::Mat object
cv::Mat camFrame;
 
	// *** loop ***
 
	// Grab the device
	capture.grab();
 
	// Retrieve desired sensor data (in this case the standard camera image)
	capture.retrieve(camFrame, CV_CAP_OPENNI_BGR_IMAGE);
 
	// Convert to texture
	GLuint tex = matToTexture(camFrame, GL_NEAREST, GL_NEAREST, GL_CLAMP);
 
	// Bind texture
	glBindTexture(GL_TEXTURE_2D, tex);
 
	// Do whatever you want with the texture here...
 
	// Free the texture memory
	glDeleteTextures(1, &tex);
 
	// *** End of loop ***
 
// Release the device
capture.release();

There's one very important issue to watch out for when using OpenCV and OpenNI together which I've commented in the code, but I'll place here as well as it can be a real deal breaker:

There appears to be a threading issue with the OpenCV grab() function where if you try to grab the device before it's ready to provide the next frame it takes up to 2 seconds to provide the frame, which it might do for a little while before crashing the XnSensorServer process & then you can't get any more frames without restarting the application. This results in horrible, stuttery framerates and garbled sensor data.

I've found that this can be worked around by playing an mp3 in the background. No, really. I'm guessing the threading of the mp3 player introduces some kind of latency which prevents the grab() function being called too soon. Try it if you don't believe me!

So just be aware that if you're using a Kinect you have to be careful with the grab() function... The source code used to create the above video is provided in full after the jump, if you're interested.

Cheers!

Read the rest of this entry »

Comments
12 Comments »
Categories
Coding, Linux
Tags
Capture, Conversion, Convert, cv::Mat, Kinect, Linux, NITE, OpenCV, OpenGL, OpenNI, Texture
Comments rss Comments rss
Trackback Trackback

Translate

Categories

Archives

Tags

3D ActionScript ActionScript 3.0 Adobe AI Ballarat Bash C++ Class Cover CS4 Effect Error Film Flash FPS GLFW Glitch GLSL Hack How-To install Java Kinect Linux Live Mash-Up Microsoft Motion mount OpenGL Particle Problem PS3 Remix Retro script Slides Sound Ubuntu Video VirtualBox Wii Windows XBox

Gamercard

OpenR3dux

Misc.

Flattr this

RSS Feed

r3dux twitter feed



“The world is a nice place, and worth fighting for.”

 - Ernest Hemmingway

rss Comments rss valid xhtml 1.1 design by jide powered by Wordpress get firefox