DirectShow ISampleGrabber: swatches upside down and color channels access

Question

I need to use MS DirectShow to capture video frames from the camera (I just want to get raw pixel data).
I was able to build a Graph / Filter network (gripper filter and ISampleGrabber) and implement a callback (ISampleGrabberCB). I get samples of the appropriate size.

However, they are always flipped (flipped vertically rather than rotated) and the color channels are in BGR order (not RGB).

I tried setting the biHeight field in BITMAPINFOHEADER to both positive and negative values, but this has no effect. According to the MSDN documentation, ISampleGrapper :: SetMediaType () ignores the format block for video in any case.

Here's what I see (recorded with a different camera, not DS), and that gives me DirectShow ISampleGrabber: "RGB" is actually red, green and blue respectively:

The sample code I'm using is a bit simplified:

// Setting the media type... AM_MEDIA_TYPE* media_type = 0 ; this->ds.device_streamconfig->GetFormat(&media_type); // The IAMStreamConfig of the capture device // Find the BMI header in the media type struct BITMAPINFOHEADER* bmi_header; if (media_type->formattype != FORMAT_VideoInfo) { bmi_header = &((VIDEOINFOHEADER*)media_type->pbFormat)->bmiHeader; } else if (media_type->formattype != FORMAT_VideoInfo2) { bmi_header = &((VIDEOINFOHEADER2*)media_type->pbFormat)->bmiHeader; } else { return false; } // Apply changes media_type->subtype = MEDIASUBTYPE_RGB24; bmi_header->biWidth = width; bmi_header->biHeight = height; // Set format to video device this->ds.device_streamconfig->SetFormat(media_type); // Set format for sample grabber // bmi_header->biHeight = -(height); // tried this for either and both interfaces, no effect this->ds.sample_grabber->SetMediaType(media_type); // Connect filter pins IPin* out_pin= getFilterPin(this->ds.device_filter, OUT, 0); // IBaseFilter interface for the capture device IPin* in_pin = getFilterPin(this->ds.sample_grabber_filter, IN, 0); // IBaseFilter interface for the sample grabber filter out_pin->Connect(in_pin, media_type); // Start capturing by callback this->ds.sample_grabber->SetBufferSamples(false); this->ds.sample_grabber->SetOneShot(false); this->ds.sample_grabber->SetCallback(this, 1); // start recording this->ds.media_control->Run(); // IMediaControl interface 

I check return types for each function and get no errors.

I am grateful for any hint or idea.

Things I've already tried:

• Setting the biHeight field to a negative value for the capture or capture filter of a sample, or both or for them, has no effect.

• Using IGraphBuilder to connect contacts is the same problem.

• Connecting contacts before changing the media type is the same problem.

• Checking whether the type of material was actually applied by the filter, requesting it again, but it seems to be applied, or at least saved.

• Interpreting an image as a common byte in reverse order (last byte first, first byte last) - then it will be flipped horizontally.

• Checking for a problem with the camcorder - when I test it using VLC (DirectShow capture), it looks normal.

Answer 1

I noticed that when I use color space rotation, the I420 disappears. In addition, most modern codecs (VP8) are used as color space I / O format I4O format.

I wrote a simple mirroring function in the I420 color space.

 void Camera::OutputCallback(unsigned char* data, int len, uint32_t timestamp, void *instance_) { Camera *instance = reinterpret_cast<Camera*>(instance_); Transport::RTPPacket packet; packet.rtpHeader.ts = timestamp; packet.payload = data; packet.payloadSize = len; if (instance->mirror) { Video::ResolutionValues rv = Video::GetValues(instance->resolution); int k = 0; // Chroma values for (int i = 0; i != rv.height; ++i) { for (int j = rv.width; j != 0; --j) { int l = ((rv.width * i) + j); instance->buffer[k++] = data[l]; } } // U values for (int i = 0; i != rv.height/2; ++i) { for (int j = (rv.width/2); j != 0; --j) { int l = (((rv.width / 2) * i) + j) + rv.height*rv.width; instance->buffer[k++] = data[l]; } } // V values for (int i = 0; i != rv.height / 2; ++i) { for (int j = (rv.width / 2); j != 0; --j) { int l = (((rv.width / 2) * i) + j) + rv.height*rv.width + (rv.width/2)*(rv.height/2); if (l == len) { instance->buffer[k++] = 0; } else { instance->buffer[k++] = data[l]; } } } packet.payload = instance->buffer; } instance->receiver->Send(packet); } 

Answer 2

My quick hack for this:

 void Camera::OutputCallback(unsigned char* data, int len, void *instance_) { Camera *instance = reinterpret_cast<Camera*>(instance_); int j = 0; for (int i = len-4; i > 0; i-=4) { instance->buffer[j] = data[i]; instance->buffer[j + 1] = data[i + 1]; instance->buffer[j + 2] = data[i + 2]; instance->buffer[j + 3] = data[i + 3]; j += 4; } Transport::RTPPacket packet; packet.payload = instance->buffer; packet.payloadSize = len; instance->receiver->Send(packet); } 

Correct in the RGB32 color space, for other color spaces this code needs to be adjusted