Cross Platform Camera API

Question

Now I am creating a video conversion filter that should convert video frames in real time. One of the key filter requirements is high performance in order to minimize the number of dropped frames during conversion.

Another requirement, which has a lower priority, but also pleasant, is to make it cross-platform (both PCs and mobile devices).

The application is built in C ++.

Now my question is:

Is there any API that is more portable and has similar or better performance characteristics than DirectShow? since DirectShow portability is limited only to Windows-based devices (PCs and Windows Mobile and CE platforms).

I also notice that, for example, using the HTC user API has much better performance than what DirectShow offers. If you want to check this out, try creating a filter in DirectShow that will multiply each color by 2 and display it in real time from the camera on the screen. Then do the same with the HTC API. Compared to the vendor API, there is an almost 4-5-fold increase in performance. Therefore, it would be very nice if the library used the implementation of the driver for a specific device, since performance is crucial when performing this conversion on a mobile device (about 500 MHz).

Answer 1

There is opencv.org , which has an extremely simple cross-platform interface for connecting to cameras, however it is a complex system designed for image processing applications and does not directly provide capture. You will need to process your frames and transfer them to a file. Also, in terms of performance, when I looked at openCV, it seems that a very old 16-bit video card on Windows is used on its side of the window :(.

You can try the framework of the Nokia QT application. They are currently developing the QT Multimedia library that does what you want. However, this is currently a beta product, and my experience (3-4 months ago) was that the Windows port still needs some work. YMMV.
You will need to install QT and then the “new QIS APIS project - Mobility” from the “Other downloads” section of qt.nokia.com/downloads/ . Depending on the time frame of your project, this may be a good option for you ... QT is, of course, a good IMHO toolkit.

Answer 2

Consider doing this in OpenGL shaders, then filters can be hardware accelerated, and they can work on mobile devices and GLES mobile devices (but they will not be fully compatible). The operations that you can (easily) perform in this way are somewhat limited, but most of the things you would like to do in filtering the video can certainly be done.