We are looking for a way to send frame metadata (for example, an identifier) with H264 encoded frames from the server to the client.
We are currently developing a remote rendering application in which both the client and the server are actively involved. The server displays a high quality image with all effects, lighting, etc. The client also has model information and displays a diffuse image, which is used when the bandwidth is too low or the images must be distorted to avoid stuttering.
For now, we encode the server-side frames using ffmpeg and transfer them from live555 to the client, which receives the rtsp stream and decodes the frames again using ffmpeg.
For our application, we need to send metadata for each frame. We want the client to tell the server where the camera is right now. Ideally, we could send the client viewing matrix to the server, display the corresponding frame and send it to the client along with our view matrix. Therefore, when the client receives the frame, we need to know exactly in which position the camera was displayed.
As an alternative, we can also mark each view matrix with an identifier, send it to the server, display a frame and mark it with the same identifier and send it back. In this case, we will again have to assign the correct matrix to the frame on the client side.
After several attempts to realize the above intention with ffmpeg, we came to the conclusion that ffmpeg does not provide the required functionality. ffmpeg provides only a predefined set of fields for metadata that either cannot store the matrix or can only be set for each keyframe, which is not often enough for our purpose.
Now we are considering using live555. So far, we have an on-demand server that receives a VideoSubsession with H264VideoStreamDiscreteFramer to contain our own FramedSource class. In this class, we download the encoded AVPacket (from ffmpeg) and send our data buffer over the network. Now we need a way to send some metadata to each frame to the client.
Do you have any ideas on how to solve this metadata problem using live555 from another library?
Thank you for your help!