All geek questions in one place

How to visualize a dynamic stream of raster images on a canvas?

I wrote an OpenGL game and I want to enable remote play of the game through the canvas element. The input is simple, but the video is complicated.

What I'm doing right now is launching the game through node.js, and in my rendering cycle I send to stdout a base64 encoded stream of raster data representing the current frame. The base64 frame is sent via websocket to the client page and rendered (painstakingly slow) in pixels. Obviously, this does not hold.

I'm starting to think about trying to create a video stream, and then I can easily display it on the canvas using the tag (ala http://mrdoob.github.com/three.js/examples/materials_video.html ).

The problem with this idea is that I don’t know enough about codecs / streams to define at a high level, if this is really possible? I’m not sure that even the codec is the part in which I need to worry about the possibility of dynamically changing content and, possibly, a few frames ahead.

Other ideas that I had:

  • Trying to create an HTMLImageElement element from a base64 frame
  • Trying to optimize the compression / redrawing areas so that the pixel throughput is much lower (it seems unrealistic to achieve such performance that I need to get 20 + fps).

Then there is always the opportunity to switch to flash ... but I would prefer to avoid this. I'm looking for some opinions about the technologies that need to be implemented, ideas?

+4
source share
2 answers

  • Why is base64 encoding data? I think you can push raw bytes on top of WebSocket

  • If you have a linear array of RGBA values ​​in the correct format, you can dump them directly into the ImageData object for later use with a single call to ctx.putImageData() .

+1
source

Try converting RGB to YCbCr color space and stream pixel values ​​as:

 Y1 Y2 Y3 Y4 Y5 .... Cb1 Cb2 Cb3 Cb4 Cb5 .... Cr1 Cr2 Cr3 Cr4 Cr5 ...

There would be many repeating patterns, so any compression algorithm would compress it better than the RGBRGBRBG sequence.

http://en.wikipedia.org/wiki/YCbCr

+2
source

Source: https://habr.com/ru/post/1340546/

More articles:


All Articles