Alternatives for generating a video stream from screenshots

Question

I work in a remote administration toy project. At the moment, I can take screenshots and control the mouse using the Robot class. Screenshots are BufferedImage instances.

First of all, my requirements: - Only the server and client. - Performance is important because the client can be an Android application.

I thought about opening two sockets, one for the mouse and system commands, and the second for the video stream.

How to convert screenshots to video stream? Should I convert them to a known video format or would it be ok to just send a series of serialized images?

Compression is another issue. According to my preliminary tests, transferring screenshots in full resolution will result in a low frame rate. I think I need at least 24 frames per second to perceive movement, so I need to both scale and compress. I could convert the BufferedImages files to jpg and then set the compression speed, but I do not want to store the files on disk, they should only live in RAM. Another option would be to serialize instances (representing an uncompressed screenshot) into a GZipOutputStream. What is the right approach for this?

Summarizing:

• If you recommend a "series of images" approach, how would you serialize them to an OutputStream socket?
• If your suggestion is to convert to a famous video format, what classes or libraries are available?

Thanks in advance.

UPDATE: my tests, client and server on the same computer
-Full serialized BufferedImages screen (only size, type and int []), no compression: 1.9 frames / s.
- Full-screen images through GZip streams: 2.6 frames per second.
- thumbnails (width 640) and GZip streams: 6.56 frames / s.
- Full screen image and RLE encoding: 4.14 fps.
-Downscaled images and RLE encoding: 7.29 fps.

Answer 1

If it were just to capture the screen, I wouldn’t compress them using the video compression scheme, most likely you don’t want lossy compression (blurry details in small text, etc. are the most common defects). To get a working "remote desktop", remember the previously sent screenshot and send only the difference to go to the next. If nothing (or very little) changes between frames, it is very effective. However, it will not work in certain situations, such as playing a video, playing a game, or scrolling through a document.

Compressing the difference between the two BufferedImage can be done using more or less complicated methods, a very simple but quite effective method is to simply subtract one image from the other (as a result of which the zeros are identical everywhere) and compress the result using a simple RLE (path length encoding )

Decreasing color accuracy can be used to further reduce the amount of data (depending on the option used, you can omit the least significant N bits of each color channel, since most GUI applications do not look as if you were reducing colors from 24 bits to 15 bits )

Answer 2

• Split screen into grid squares (or stripes)
• Only send a grid square if it differs from the previous one

// server start

 sendScreenMetaToClient(); // width, height, how many grid squares ... // server loop ImageBuffer[] prevScrnGrid while(isRunning) { ImageBuffer scrn = captureScreen(); ImageBuffer[] scrnGrid = screenToGrid(scrn); for(int i = 0; i < scrnGrid.length; i++) { if(isSameImage(scrnGrid[i], prevScrnGrid[i]) == false) { prevScrnGrid[i] = scrnGrid[i]; sendGridSquareToClient(i, scrnGrid[i]); // send the client a message saying it will get grid square (i) then send the bytes for grid square (i) } } } 

Do not send serialized java objects, just send image data.

 ByteArrayOutputStream imgBytes = new ByteArrayOutputStream(); ImageIO.write( bufferedImage, "jpg", imgBytes ); imgBytes.flush(); 

Answer 3

Firstly, I could only offer to capture a small part of the screen, and not to zoom out and possibly lose information, perhaps with something like a sliding window that can be moved by pressing the edges with the cursor. This is really just a small design proposal.

As for compression, I would think that a series of images would not be compressed separately, as well as with an acceptable video compression scheme, especially since frames are likely to remain consistent between captures in this scenario.

One option is to use Xuggle, which is capable of capturing the desktop via Robot in a number of afaiu video formats, but I cannot say if you can stream and decode with this.

You can also use this to capture jpeg and convert them.

Streaming video seems a bit more complicated.

In addition, it seems that the abandoned Java Media Framework supports this functionality.

My knowledge in this area is not fantastic tbh, so I'm sorry if I spent your time, but it seems that some more useful information about the possibility of using Xuggle as a screenshot was compiled here . It also appears to be related to their own notes on existing approaches.

If you do not need pure Java, I would think that all this will be much simpler, just using the interface with its own screen capture tool ...

Perhaps it would be easiest to post the video as a series of jpegs in the end! You can always implement your own compression scheme if you feel a little crazy ...

Answer 4

I think you described a good solution in your question. Convert images to jpeg, but do not write them as files to disk. If you want this to be a well-known video format, use M-JPEG. M-JPEG is a jpeg frame stream in standard format. Many digital cameras, especially older ones, save video in this format.

You can get some information on how to play the M-JPEG stream from this question: Android and MJPEG

If network bandwidth is a problem, then you will want to use an inter-frame compression system such as MPEG-2, h.264 or the like. This requires much more processing than M-JPEG, but is much more efficient.

Answer 5

If you are trying to get 24fps video, then there is no reason not to use modern video codecs. Why try to recreate this wheel?

Xuggler works great for h264 video encoding and sounds as if it would suit your needs well.