Video Library

Introduction

In Processing 2.0, we've made the switch to a gstreamer-based video library (excising the Apple horror show that was QuickTime for Java).

There are many tales to tell of the quirky and unique beast that is gstreamer, and this document seeks to tell them.

Build Process

The build process is long and hard. Someday this section might detail the difficult journey.

Listing Capture Devices

The Capture.list() method operates differently on 2.0 than previous releases. In gstreamer, there's a concept of actual supported resolutions (and frame rates) for capture, so Capture.list() has been expanded to use all of that information.

OS X, however, thinks different, and allows any capture dimensions to be valid. To keep things in line with Windows and Linux, we're forced to pre-roll each of the cameras with no/default settings, which seems to give us the native resolution of the camera. Then that size is cut in half a few times over (until sizes get below 100 pixels) to produce a few synthetic sizes that can be reported via Capture.list().

The name-based selection of devices is half-supported in gstreamer, and looks to be going away, so we'll need our own code to keep this feature.

Some notes from Andrés extracted from a bug report

As a reference, here is the link to the bugzilla entry about the qtkitvideosrc element, which is used by the video library to do capture under OSX: https://bugzilla.gnome.org/show_bug.cgi?id=661348

Hopefully they will include my patches into the main line, and won't remove the property probe interface (needed to get the camera names) or, in the case they do, will replace with an analogous API.

I just got the confirmation from the gstreamer developers that the property probe interface has been removed from the upcoming 1.0 release, although they say there will be a replacement at some later point.

Until that replacement comes in, we might need to call the functions in QTKit and DirectShow that gives you the list of currently available devices directly from the video library. This is, actually, very simple code. for QTKit:

NSArray *devices = [QTCaptureDevice inputDevicesWithMediaType:QTMediaTypeVideo];

and for Direct Show:

HDEVINFO devinfo = SetupDiGetClassDevsW (&KSCATEGORY_VIDEO, NULL, NULL, DIGCF_PRESENT | DIGCF_DEVICEINTERFACE);

The advantage of the property probe interface in gstreamer is that we got it for free through the gstreamer-java bindings. Now we will need to add our own binding JNA code to call them directly from Processing.

Another thing is that the property probe for camera names never worked on Linux, but I found that on Linux appears to be a mechanism to retrieve names:

http://www.freedesktop.org/software/systemd/gudev/

http://gudev.sourceforge.net/ (seems like an outdated link)

so it would be a matter to write the appropriate wrapper for it as well:

http://www.freedesktop.org/software/systemd/gudev/GUdevEnumerator.html