“Fun” with conversion: PXCImage -> OpenCV -> OpenGL

So in order to do some serious image processing with the Intel PCSDK, you need OpenCV, and to do that, you need to convert your pictures from Intel PCSDK (PXCImage::ImageData) to OpenCV (IplImage and cv::Mat)and maybe to OpenGL textures, if you want to display them.

I’ve seen all kinds of conversions over the internet, and all of the included nasty pitches and borders and flipping images – but I think I’ve found something logical and useable.

First, let’s work on color pictures.
You need your usual elements:

gl::Texture mLabelMap;
PXCImage::ImageData mCImageData; //Color Image data..)

Where your acquiring (in CINDER, this is probably in your update()), call this:

if(mCImage->AcquireAccess(PXCImage::ACCESS_READ, &mCImageData)>=PXC_STATUS_NO_ERROR)
{
IplImage* colorimg = cvCreateImageHeader(cvSize(640, 480), 8, 3);
cvSetData(colorimg, (uchar*)mCImageData.planes[0], 640*3*sizeof(uchar));
cv::Mat image(colorimg);
/*Now you can whatever you want with OpenCV*/
mLabelMap =gl::Texture(fromOcv(image)); //Here's how to get to OpenGL
..

Displaying this (in the draw() function if you are using CINDER), is easy as pie:


if(mLabelMap)
gl::draw(mLabelMap, Rectf(0,0,mCLW,mCLH));

This was inspired by this forum post: http://software.intel.com/en-us/forums/topic/375707
where you can also read the parameters for the depth picture and the UV image:


IplImage* depthimg = cvCreateImageHeader(cvSize(320, 240), 16, 1);
IplImage* UVimg = cvCreateImageHeader(cvSize(320, 240), 32, 2);
..
PXCImage *colorImage = mCapture->QueryImage(images, PXCImage::IMAGE_TYPE_COLOR);
PXCImage *depthImage = mCapture->QueryImage(images, PXCImage::IMAGE_TYPE_DEPTH);
...
depthImage->AcquireAccess(PXCImage::ACCESS_READ, &depthImageData)
cvSetData(depthimg, (short*)depthImageData.planes[0], depthsize.width*sizeof(short)*1);
cvSetData(UVimg, (float*)depthImageData.planes[2], depthsize.width*2*sizeof(float));

The rest is the same.

With a bit of trial/API studying, I also found a solution for the blob image, use this:


if(mGesture->QueryBlobImage(PXCGesture::Blob::LABEL_SCENE, 0, &mImage)>=PXC_STATUS_NO_ERROR)
if(mImage)
if(mImage->AcquireAccess(PXCImage::ACCESS_READ, &mImageData)>=PXC_STATUS_NO_ERROR)
{
IplImage* blobimg = cvCreateImageHeader(cvSize(320, 240), 8, 1);
cvSetData(blobimg, (uchar*)mImageData.planes[0], 320*sizeof(uchar));
cv::Mat image(blobimg);
/*Do whatever you want with OpenCV*/
mLabelMap =gl::Texture(fromOcv(image));

(By the way, if you want to access the labels for your hand (for example for thresholding), you need the following code:)


PXCGesture::Blob bdata;
mGesture->QueryBlobData(PXCGesture::Blob::LABEL_SCENE,0,&bdata);
int num=(int)bdata.labelRightHand;
int num2=(int)bdata.labelLeftHand;

 

Edit: 6.3.2014, Updated a line for depthimage, see discussion below. You can also use the sample code there.

Advertisements

15 comments

  1. Works for me, thaks

  2. Could you actually post the entire code please? I’m a beginner and can’t follow.

    1. What’s your problem? Have you installed CINDER? Can you get the samples running?

      1. No, I don’t have CINDER. My problem is that I need a way to convert PXCImage to OpenCV Mat format. I’ve come across your blog via google. It’s the only relevant entry I could find. Could you provide the whole code please?

  3. It’s not that I don’t want to give it to you – it was just a throwaway thing and I don’t have a running sample anywhere near.

    What kind of PXCImage do you have? Depth? Color? Can you display it?
    What kind of errors message are you getting?

    The process is pretty much always the same:
    First, you create an IplImage, which is a struct that contains information about the image data as well as about depth, size, ..
    For example, this would be a color image of 640*480 with 8 bit depth and 3 planes.

    IplImage* colorimg = cvCreateImageHeader(cvSize(640, 480), 8, 3);

    Then you tell the program where your data is (from your PXImage).
    I wrote this blog post, because this is actually the difficult step – PXImage stores data differently than OpenCV, so you have to do this properly. For a color picture it would be:

    cvSetData(colorimg, (uchar*)mCImageData.planes[0], 640*3*sizeof(uchar));

    As a final step, you create a cv::Mat, which just means that you are extracting the pure data to work with.
    cv::Mat image(colorimg);

    1. First of, thanks for your rapid responses! Anyways, with your explanation, it has become much more clearer and I’ve put a little prototype together, which gets the images from my creative camera, transforms them into matrix format, and outputs them with the imshow command of openCV. However, I get an access violation on the code line:

      img->AcquireAccess(PXCImage::ACCESS_READ, &mCImageData)

      Where:

      PXCImage *img = pipeline.QueryImage(PXCImage::IMAGE_TYPE_COLOR);

      Whole code:

      void convert() {
      UtilPipeline pipeline;
      pipeline.EnableImage(PXCImage::COLOR_FORMAT_DEPTH);
      pipeline.Init();
      UtilRender depth_render(L”Depth Stream”);
      while (true) {
      if (!pipeline.AcquireFrame(true))
      break;

      PXCImage *img = pipeline.QueryImage(PXCImage::IMAGE_TYPE_COLOR);
      PXCImage::ImageData mCImageData;
      if (img->AcquireAccess(PXCImage::ACCESS_READ, &mCImageData) >= PXC_STATUS_NO_ERROR) {
      IplImage* colorimg = cvCreateImageHeader(cvSize(640,480), 8, 3);
      cvSetData(colorimg, (uchar*)mCImageData.planes[0], 640*3*sizeof(char));
      Mat image(colorimg);
      imshow(“Example”, image);

      }

      if (!depth_render.RenderFrame(img))
      break;

      pipeline.ReleaseFrame();

      }

      pipeline.Close();
      }

  4. Let’s do it step by step.
    This is not an issue with converting – it happens sooner.
    As far as I see ( don’t have a compiler here..), you are setting up the pipeline for a depth image, but then aquiring a color image. So as a first step, try to enable the pipeline to capture color pics, so add this line before your init:

    pipeline.EnableImage(PXCImage::COLOR_FORMAT_RGB24,640,480);

    Comment everything that doesn’t work and try to render this with the UtilPipeline. Then we are sure you’re getting proper color image data, and can move on.

    1. Wonderful! It now finally works. You’ve saved me a lot of hassle. Thanks a lot! I’m now trying to get the depth stream to work, however, I think I’m making a mistake with the step size. You write:

      cvSetData(depthimg, (short*)depthImageData.planes[0], depthsize.width*sizeof(short)*3);

      Wouldn’t depthsize be equal to 320 in the example of your blog?

      1. The error I’m getting is again an Access Violation.

  5. Yes, depthsize is 320.

    Where are you getting an access violation? Again, can you output the depth image without any processing? Have you created the depthimg properly?

    1. Yes, I can output the depth images without processing with the PercC library functions (with RenderFrame). I get an access violation at imshow(“Example”, final), where final is the matrix depicting PXCImage in OpenCV Mat format. The first few frame actually display a sort of distorted picture. After about 4 loops, the memory access violation occurs. Here is my code:

      void convert() {
      UtilPipeline pipeline;
      pipeline.EnableImage(PXCImage::COLOR_FORMAT_DEPTH, 320, 240);
      pipeline.Init();
      while (true) {

      if (!pipeline.AcquireFrame(true))
      break;

      PXCImage *depth = pipeline.QueryImage(PXCImage::IMAGE_TYPE_DEPTH);
      PXCImage::ImageData dImageData;

      if (depth->AcquireAccess(PXCImage::ACCESS_READ, &dImageData) >= PXC_STATUS_NO_ERROR) {
      IplImage* depthimg = cvCreateImageHeader(cvSize(320,240), 16, 1);
      cvSetData(depthimg, (short*)dImageData.planes[0], 320*sizeof(short)*3);
      Mat final(depthimg);
      imshow(“Example 2”, final);
      }

      waitKey(0);

      pipeline.ReleaseFrame();

      }

      pipeline.Close();
      }

      waitKey is there just so I could see the individual frames.

  6. I’m off into wildly speculative remote debugging right now 🙂
    A distorted image means that we are accessing the data differently than it is presented “for real”. Maybe they changed specs (I worked with a very beta version of the thing), maybe I changed something from another project. Can’t really remember.

    It would be interesting and helpful to see that image, but let’s do what we can.
    Let’s look at this line:
    cvSetData(depthimg, (short*)dImageData.planes[0], 320*sizeof(short)*3);

    The last part is the so-called step-size. That means the length from the start of one row to the next, in the buffer of pixels.
    If you are getting memory access errors, you are going too far.
    So let’s go less far, and I’m geussing
    it’s cvSetData(depthimg, (short*)dImageData.planes[0], 320*sizeof(short));
    because it would make more sense if the depth image doesn’t have 3 byte values.
    It might have 2 byte values, however, but let’s try this for starters.

    1. Aaaaand you speculated correctly! 😀 It works, however the colors are inverted. See the pic here:

      Is there anything I can do to avoid that during conversion or will I just have to use some OpenCV features to invert it back?

      1. Thanks for the feedback!

        Personally, I’d just use OpenCV.
        There might be something inside the PCSDK, (I’d go looking for some masks or bitshifts), but inverting a 320*240 pic is not going to cost you any significant computing power.
        (And if that mattered, you should use the SoftKinetic SDK anyway.)

  7. Alright, thanks again for everything! 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: