How to parse through an infinite scroll page (e.g. Wallbase.cc/search/sky) using Python?

Question

Not sure if there is anything with Mechanize or BeautifulSoup that could help. Any suggestions would be greatly appreciated!

Answer 1

The mechanism and the fine soup cannot go into javascript used for endless scrolling.

Selenium can.

Also, if you want to view ajax requests using infinite scroll, you will see a request to send http://wallbase.cc/search/160 with the request data:

 query:sky board:123 res_opt:eqeq res:0x0 aspect:0 nsfw_sfw:1 nsfw_sketchy:0 nsfw_nsfw:0 thpp:32 orderby:relevance orderby_opt:desc 

160 corresponds to the image range, so the request before it was wallbase.cc/searc/128 .