r3dux.org

The Leap is a new 3D input motion controller in a similar vein to the Kinect, but with apparently about 200x the resolution – how amazing is that?

To quote the Leap people:

Leap represents an entirely new way to interact with your computers. It’s more accurate than a mouse, as reliable as a keyboard and more sensitive than a touchscreen. For the first time, you can control a computer in three dimensions with your natural hand and finger movements.

I’ve applied for a free dev-kit, so fingers-crossed! Worst case though, the sensor’s only meant to be around the $70 mark when they come out.

Would so love to work with this tech for my Ph.D research… looks like it solves a lot of problems right off the bat. Awesome!

Looks like I might not have to use the OpenNI proprietary blob for skeleton tracking in my research project afterall… Ties in with Gnome 3 nicely as well!

Will have to get experimenting with that bad boy! =D

Voice recognition is only ever going to get better (check out CMU Sphinx if you want to play with speech rec. for yourself), but at the moment… um, it’s not quite up to scratch:

Okay, so he’s putting in on a bit for the lulz, and if he’d just said “where’s the nearest pub?” he’d probably get a useful answer – but that’s half the battle with speech recognition. At the moment we have to adapt our speech to the software, but in the future I don’t think we’ll have to at all.

At least for the time being, as Penny Arcade put it, it’s all going to be a bit like this:

Best keep your deer-combs handy.

I’m working on using OpenCV to get Kinect sensor data via OpenNI, and needed a way to get a matrix (cv::Mat) into an OpenGL texture – so I wrote a function to do just that – woo! Apologies in advance for the terrible juggling

The function used to perform the sensor data to texture conversion is:

// Function turn a cv::Mat into a texture, and return the texture ID as a GLuint for use
GLuint matToTexture(cv::Mat &mat, GLenum minFilter, GLenum magFilter, GLenum wrapFilter)
{
	// Generate a number for our textureID's unique handle
	GLuint textureID;
	glGenTextures(1, &textureID);
 
	// Bind to our texture handle
	glBindTexture(GL_TEXTURE_2D, textureID);
 
	// Catch silly-mistake texture interpolation method for magnification
	if (magFilter == GL_LINEAR_MIPMAP_LINEAR  ||
	    magFilter == GL_LINEAR_MIPMAP_NEAREST ||
	    magFilter == GL_NEAREST_MIPMAP_LINEAR ||
	    magFilter == GL_NEAREST_MIPMAP_NEAREST)
	{
		cout < < "You can't use MIPMAPs for magnification - setting filter to GL_LINEAR" << endl;
		magFilter = GL_LINEAR;
	}
 
	// Set texture interpolation methods for minification and magnification
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, minFilter);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, magFilter);
 
	// Set texture clamping method
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, wrapFilter);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, wrapFilter);
 
	// Set incoming texture format to:
	// GL_BGR       for CV_CAP_OPENNI_BGR_IMAGE,
	// GL_LUMINANCE for CV_CAP_OPENNI_DISPARITY_MAP,
	// Work out other mappings as required ( there's a list in comments in main() )
	GLenum inputColourFormat = GL_BGR;
	if (mat.channels() == 1)
	{
		inputColourFormat = GL_LUMINANCE;
	}
 
	// Create the texture
	glTexImage2D(GL_TEXTURE_2D,     // Type of texture
	             0,                 // Pyramid level (for mip-mapping) - 0 is the top level
	             GL_RGB,            // Internal colour format to convert to
	             mat.cols,          // Image width  i.e. 640 for Kinect in standard mode
	             mat.rows,          // Image height i.e. 480 for Kinect in standard mode
	             0,                 // Border width in pixels (can either be 1 or 0)
	             inputColourFormat, // Input image format (i.e. GL_RGB, GL_RGBA, GL_BGR etc.)
	             GL_UNSIGNED_BYTE,  // Image data type
	             mat.ptr());        // The actual image data itself
 
	// If we're using mipmaps then generate them. Note: This requires OpenGL 3.0 or higher
	if (minFilter == GL_LINEAR_MIPMAP_LINEAR  ||
	    minFilter == GL_LINEAR_MIPMAP_NEAREST ||
	    minFilter == GL_NEAREST_MIPMAP_LINEAR ||
	    minFilter == GL_NEAREST_MIPMAP_NEAREST)
	{
		glGenerateMipmap(GL_TEXTURE_2D);
	}
 
	return textureID;
}

You can then use the above function like this:

// Create our capture object
cv::VideoCapture capture( CV_CAP_OPENNI );
 
// Check that we have actually opened a connection to the sensor
if( !capture.isOpened() )
{
	cout < < "Cannot open capture object." << endl;
	exit(-1);
}
 
// Create our cv::Mat object
cv::Mat camFrame;
 
	// *** loop ***
 
	// Grab the device
	capture.grab();
 
	// Retrieve desired sensor data (in this case the standard camera image)
	capture.retrieve(camFrame, CV_CAP_OPENNI_BGR_IMAGE);
 
	// Convert to texture
	GLuint tex = matToTexture(camFrame, GL_NEAREST, GL_NEAREST, GL_CLAMP);
 
	// Bind texture
	glBindTexture(GL_TEXTURE_2D, tex);
 
	// Do whatever you want with the texture here...
 
	// Free the texture memory
	glDeleteTextures(1, &tex);
 
	// *** End of loop ***
 
// Release the device
capture.release();

There's one very important issue to watch out for when using OpenCV and OpenNI together which I've commented in the code, but I'll place here as well as it can be a real deal breaker:

There appears to be a threading issue with the OpenCV grab() function where if you try to grab the device before it's ready to provide the next frame it takes up to 2 seconds to provide the frame, which it might do for a little while before crashing the XnSensorServer process & then you can't get any more frames without restarting the application. This results in horrible, stuttery framerates and garbled sensor data.

I've found that this can be worked around by playing an mp3 in the background. No, really. I'm guessing the threading of the mp3 player introduces some kind of latency which prevents the grab() function being called too soon. Try it if you don't believe me!

So just be aware that if you're using a Kinect you have to be careful with the grab() function... The source code used to create the above video is provided in full after the jump, if you're interested.

Cheers!

Read the rest of this entry »

New on-rails-rhythm-shooter-thingy Child of Eden (from Tetsuya Mizuguchi the same guy who designed Rez, no less) is looking pretty darn nifty, especially with Kinect controls. Though quite how long you’ll last without your arms getting tired is quite another matter.

If you’ve never played or heard of Rez – it has to be one of the most beautiful and original games ever made. Originally for the Dreamcast, it has a HD remake on the 360 (via XBox Live Arcade), and is just great. Also, it has one of the easiest control schemes ever, meaning pretty much anyone can play it even if they’re not used to game controllers. You move the targeting reticule around (up/down/left/right), and hold down or hammer the fire button. If you’re holding the button down, anything that passes through the reticule is targeted (up to 8 things, or 8 shots on the same thing, or any combination), and when you release the button, up to eight shots shoot off and hit stuff. All perfectly timed to each levels unique techno/house/ambient soundtrack. Brilliant!

See? If Child of Eden is Rez 2.0, then I definitely want some

Also, and I only just noticed this – but I posted about the track California Soul by Marlena Shaw (or a remix thereof) the other day – and guess what the music in the latter half of the Rez video samples? I knew I’d heard it somewhere before! How’s that for serendipity?