How can I extract text content from graphical applications on Linux?

Question

I want to extract text content from graphical applications, here are 2 examples:

example 1:

Suppose I opened firefox and enter url: www.google.com

How can I extract the string "www.google.com" from firefox using my own application?

Example 2:

open calculator (using gcalctool), then enter 1 + 1

How can I extract the string "1 + 1" of the calculator from my own program?

In short, I want to find out if there is a way to extract text content from any widget in a GUI application.

thanks

Answer 1

I do not think that there is a general way to do this, at least not very elegant.

Some inelegant ideas:

You might be able to modify the X window system or even some toolkit infrastructure to extract what appears in certain windows as text.

You can take a screenshot and use the OCR library to convert pixels to text for interesting areas.

You could recompile the applications of interest to add some kind of mechanism to ask them questions.

You can use something like xtest to input events that highlight an area of ​​interest and copy them to the clipboard.

Answer 2

I believe that firefox and gcalctool are for examples only, and you just want to know how to transfer the output of one application to another application.

There are many ways to do this on Linux, for example:

pipes

application1 | application2 

btw here is the Firefox command line guide if you want to run firefox on Ubuntu with a url. eg:

 firefox "$url" 

where $ url is a variable whose value may be www.mozilla.org

Answer 3

That sounds complicated. Suppose you are using X11, you can easily get a window image (see "Man xwd"); however, there is no easy way to get to the text if it is not selected and therefore copied to the clipboard.

Alternatively, if you only want to capture user input, it is also very easy to do this by activating the X11 record extension: put this in your / etc / X 11 / xorg.conf:

 Section "Module" Load "record" #Load other modules you need ... EndSection 

although it may be difficult to use, see the sample code to expand the Xorg / X11 record failed