Using JSoup to clean up Google results

Question

I am trying to use JSoup to clear Google search results. This is currently my code.

public class GoogleOptimization { public static void main (String args[]) { Document doc; try{ doc = Jsoup.connect("https://www.google.com/search?as_q=&as_epq=%22Yorkshire+Capital%22+&as_oq=fraud+OR+allegations+OR+scam&as_eq=&as_nlo=&as_nhi=&lr=lang_en&cr=countryCA&as_qdr=all&as_sitesearch=&as_occt=any&safe=images&tbs=&as_filetype=&as_rights=").userAgent("Mozilla").ignoreHttpErrors(true).timeout(0).get(); Elements links = doc.select("what should i put here?"); for (Element link : links) { System.out.println("\n"+link.text()); } } catch (IOException e) { e.printStackTrace(); } } } 

I'm just trying to get the name of the search results and fragments under the heading. So yes, I just don’t know which element to look for in order to clear them. If anyone has a better way to clean up Google using java, I would like to know.

Thanks.

Answer 1

Here you go.

 public class ScanWebSO { public static void main (String args[]) { Document doc; try{ doc = Jsoup.connect("https://www.google.com/search?as_q=&as_epq=%22Yorkshire+Capital%22+&as_oq=fraud+OR+allegations+OR+scam&as_eq=&as_nlo=&as_nhi=&lr=lang_en&cr=countryCA&as_qdr=all&as_sitesearch=&as_occt=any&safe=images&tbs=&as_filetype=&as_rights=").userAgent("Mozilla").ignoreHttpErrors(true).timeout(0).get(); Elements links = doc.select("li[class=g]"); for (Element link : links) { Elements titles = link.select("h3[class=r]"); String title = titles.text(); Elements bodies = link.select("span[class=st]"); String body = bodies.text(); System.out.println("Title: "+title); System.out.println("Body: "+body+"\n"); } } catch (IOException e) { e.printStackTrace(); } } } 

Also, to do this on my own, I would suggest using chrome. You just right-click on what you want to clear and proceed to check the item. This will lead you to the exact place in html where this element is located. In this case, you first want to find out where the root of all the result lists is. When you find this, you want to specify an element and, preferably, a unique attribute to find it. In this case, the root element

 <ol eid="" id="rso"> 

Below you will see a list of lists starting with

 <li class="g"> 

This is what you want to put in your array of source elements, then for each element you want to find a place where there is a heading and body. In this case, I found the name under

 <h3 class="r" style="white-space: normal;"> 

element. Thus, you will search for this item in each listing. The same goes for the body. I found that the body is under so that I was looking for it using the .text () method, and it returned all the text under this element. The key is to ALWAYS try to find an element with a source attribute (using the class name ideally). If you don’t do this and are just looking for something like a “div”, it will search the entire page for ANY element containing the div and return it. This way you get more results than you want. Hope this explains it well. Let me know if you have more questions.