How to read pixel values ​​of a video?

Question

I recently wrote C programs for processing BMP image images, I had to read the pixel values ​​and process them, it was very simple, I followed the contents of the BMP header, and I could get most of the BMP image information.

Now the challenge is video processing (Frame by Frame), how can I do this? How can I read the headers of continuous streams of image frames in a video clip? Or, for example, the mpeg format will also have a universal title, after reading which I can get information about the entire video and after the title, all data will be only pixels.

Hope I could pass on.

Does anyone have experience video processing?

Any books or tutorial links would be very helpful.

Answer 1

A video stream, such as MPEG, consists of several frames, depending on (obviously) its duration and frame rate. To read a pixel, you must start with the so-called Intra frame, which is independent of the previous frame in the stream. Any consecutive frame is a frame that temporarily depends on its previous frame, so to get its pixel you need to decode the stream from Intra to the desired frame.
Note that in a typical case, an intra-frame is periodically introduced to give the decoder a way to synchronize with the stream. This is very useful in a context where errors may occur.
What you want to do is not an easy job. You must use an MPEG decoder and then modify the frame before executing it if you want to perform post-processing, such as a filter or another.
I suggest you study video coding, and you can find a lot of material starting with standard MPEG.

Answer 2

I would recommend watching FFMpeg . He has a command line utility that can capture frames from a movie and drop them onto an image like a JPG. You can then modify the existing reader to process JPG files (just use something like libjpeg to decode the JPG into the buffer of the original pixel).

Alternatively, you can use the FFMpeg API (C, Python, etc.) and do a programmatic capture of the frame and look at the pixels when moving through the video. Video formats are complex, so if you do not want to begin to understand all the different codecs, you might want to grab the library to decode the pixel buffer buffer.

Answer 3

MPEG 1/2/4 video is much more difficult to process than bitmap images because they are compressed. With raster data, you have actual color values ​​stored directly in the file. In MPEG or JPEG, in this case, the color information goes through numerical conversions before being written to a file. These include

• RGB → YUV 420P (chroma subsampling)
• Discrete cosine transform
• Weighted Quantization
• zig-zag order
• differential coding
• variable length coding (Huffman-like)

All this means that there is no easy way to analyze pixel data from a file. You need to either study every detail of the standard, or write your own decoder, or use some video decoding library, such as ffmpeg, to do this job. ffmpeg can convert your video to still images ( see answers to this recent question ). In addition, you can directly interact with ffmpeg libraries (libavformat and libavcodec). See the answers to this question for good lessons.