This is extremely difficult. I tried to do this for several years. I ultimately succeeded, but any solution would include both commercial and technical efforts.
Update March 2015
Most of the material below is no longer relevant. Now, after all these years, the package is android.media.projection https://developer.android.com/reference/android/media/projection/package-summary.html which finally allows you what you need!
Capturing screen images of your own application
For completeness, I want to include your own comment so that you can capture an image of your own application using Bitmap.createBitmap(rootview.getDrawingCache()); and similar mechanisms.
Capture the screen of another application when you are in the background
Using permission READ_FRAMEBUFFER
Firstly, you are right that a regular application cannot use the READ_FRAMEBUFFER permission, because it is a “signature” level. This means that you must be signed with the same key as the Android system ROM in order to be able to take such a screenshot.
I thought it was a little sad, so back in 2009 I made a request for an open source Android project to ask him to open 1 . Response from Dianne Hackborn, Android Architect:
Um, no. Absolutely positive no.
So then everything went well! Therefore, this permission is still a signature level to this day.
If you have this permission, you can call captureScreen member of ISurfaceComposer 2 . You will need to write your own code to access this feature using the Android NDK, as well as some undocumented APIs. However, this is possible.
Inside the Android graphics subsystem, the glReadPixels call is glReadPixels to extract pixels from the GPU back to the CPU. (The GPU is used for most compositions on Android. In fact, Android 4.0+ supports additional hardware components, and Surface Flinger needs to do even more work to get those pixels back to the CPU.)
This call works beautifully, with the exception of a few minor issues:
- The risk of using an unsupported API that could break at any time;
- The meaning of the call in C ++
- This causes the GPU pipelines to stop, which can disrupt the operation of the GPUs, but it does not actually cause problems.
- It uses large bandwidth from GPU to CPU. This is sometimes problematic because memory architectures are designed to send data in the opposite direction. However, it seems to me that all modern Android chipset architectures directly exchange memory between the GPU and the CPU, with the exception of one (maybe Broadcom? - I don’t remember), where this can lead to a very slow mechanism.
... and one big problem ...
- Most importantly, as a regular application writer, you cannot even call this API because of the required permissions at the subscription level.
However, on most Android devices you can get 10 frames per second from this. However, this API actually supports scaling the resulting image on GPU hardware, so if you're smart, you can pre-scale the image to the right size before the pixels even get on the CPU. Thus, it can be extremely high performance.
Note that you, as an application writer, cannot call glReadPixels because you do not have access to the corresponding OpenGL context. He belonged to a superficial flinger.
Using /dev/graphics/fb0 and similar
Some of them are trying to read these Linux device files that represent the framebuffer. However, there are three problems:
- You need root.
- Sometimes they are not even there.
- Often they do not represent the actual image on the screen. Remember on Android that graphic elements are built on the GPU. Therefore, there is no reason why the CPU should have access to a copy of the full composite image on the screen, and often this is not so. This file sometimes contains gaps (at best) and an image of garbage (at worst). Interestingly, some of the root phone tools use this method, which I think is a mistake. If you have root, you by definition have all permissions on Android and therefore you can call the above
captureScreen API to get the right image.
Using hardware partners
Now we find ourselves in solutions that require commercial action.
Talking with Android chipset manufacturers is often a solution. Because they design the hardware, they have access to the framebuffer - and they can often provide libraries that completely exclude the Android permissions model by simply accessing their custom kernel drivers directly.
If you are targeting a specific phone model, this is often a good way forward. Of course, the likelihood that you will need to cooperate with the manufacturer of phones, as well as with the manufacturer of silicon.
Sometimes this can provide outstanding results. For example, I heard that on some hardware you can directly connect the phone’s hardware framebuffer to the hardware video encoders of the phone’s hardware H.264 and get a pre-encoded video stream that is on the phone’s screen. Fine. (Unfortunately, I know that this is possible on TI OMAP chips, which are gradually being withdrawn from the phone market 3 ).
Using Security Holes
Android uses its permission model tightly and has several security holes. However, Android OEMs can sometimes be more careless.
For example, the main OEM, whose name begins with S, has implemented a way to capture the screen by pressing a key. It saves it in a read-only file on the SD card. Hypothetically, you might find what intercepts these keys and see how it works. Perhaps you could do something like this.
And perhaps there is a way for another major OEM, whose name also starts with S.
No, I will not go into details in this section. To understand how to do this, I need to have reverse software, and this may be illegal. Good luck, however.
Work with phone manufacturers
As described above, phone manufacturers have access to an API that works. And phone manufacturers have the required signature permissions.
So all you have to do is arrange for your software to be signed by the phone manufacturer.
It is, however, difficult. By signing the software, the phone manufacturer guarantees its quality - so they should check your source code. Also, due to the nature of Android - if they sign the software, they must be the ones who distribute it. You cannot put it on the market if it is signed with a different signature.
However, OEM should not include it in ROM - they can still distribute it in the Android market. But you can’t.
A good solution would be if each vendor signed a small library, which the shared SDK could then access. It leads me to ...
Work with software partners who have already decided this
I know a lot about this because I worked at RealVNC. We have worked with all major Android phone providers to access these signature-level APIs. I cannot overestimate the many, many years of effort (commercially and technically) necessary to achieve this. Some of the OEMs have published this work - for example 4 .
I no longer work at RealVNC, so I have nothing to gain from advertising my software. But if you really want to capture the screen on multiple Android devices, you can ask them to reuse your remote control service or Android VNC SDK 5 . This is not open-source, so you should count on payment, and believe me, this is fair enough, given the epic efforts involved in working with all of these Android OEMs.
In the interest of balance, I must point out that other manufacturers have also worked with phone manufacturers on this - for example, Soti. But I believe that they all offer specific device management solutions, rather than a common SDK for remote control / events.
For USB
Another option is the adb daemon, which listens for USB connection debugging, has slightly more privileges than a regular application, so it can capture the screen (you can see its image using the ddms tool). If you can run any command using adb , you can also get these privileges (according to the library previously bound to the android screenshot).
Contribute to Android Open Source Project
In the end, this problem reduced me to dust, and I went to greener pastures, which did not imply an attempt to squeeze pixels from Android phones.
Before I left RealVNC, we tried to bring these APIs back into the Android open source project. This time we got a more positive reaction 6 , in short, it was suggested that our approach to security was almost right, but that the graphics system was too cluttered to accept our corrections. Well, the great news is that the graphics system is no longer in turmoil - in fact, it now has a captureScreen API, which means that no changes to the graphics system are required. Therefore, it is possible to introduce a new security mechanism for AOSP around this API, which finally solves this problem.
Good luck