All geek questions in one place

How to clear a web page with dynamic content added by JavaScript?

I am trying to clear this web page , it has lazy loading, when we scroll it, it loads. Using Nokogiri, I can clear the start page, but not the rest of the page loaded after scrolling.

+4
source share
1 answer

To get a lazy loaded page, cancel the following pages:

http://www.flipkart.com/mens-footwear/shoes/casual-shoes/pr?p%5B%5D=sort%3Dpopularity&sid=osp%2Ccil%2Cnit%2Ce1f&start=31&ajax=true http://www.flipkart.com/mens-footwear/shoes/casual-shoes/pr?p%5B%5D=sort%3Dpopularity&sid=osp%2Ccil%2Cnit%2Ce1f&start=46&ajax=true http://www.flipkart.com/mens-footwear/shoes/casual-shoes/pr?p%5B%5D=sort%3Dpopularity&sid=osp%2Ccil%2Cnit%2Ce1f&start=61&ajax=true ... 
 require 'rubygems' require 'nokogiri' require 'mechanize' require 'open-uri' number = 1 while true url = "http://www.flipkart.com/mens-footwear/shoes" + "/casual-shoes/pr?p%5B%5D=sort%3Dpopularity&" + "sid=osp%2Ccil%2Cnit%2Ce1f&start=#{number}&ajax=true" doc = Nokogiri::HTML(open(url)) doc = Nokogiri::HTML(doc.at_css('#ajax').text) products = doc.css(".browse-product") break if products.size == 0 products.each do |item| title = item.at_css(".fk-display-block,.title").text.strip price = (item.at_css(".pu-final").text || '').strip link = item.at_xpath(".//a[@class='fk-display-block']/@href") image = item.at_xpath(".//div/a/img/@src") puts number puts "#{title} - #{price}" puts "http://www.flipkart.com#{link}" puts image puts "========================" number += 1 end end
+4
source

Source: https://habr.com/ru/post/1501052/

More articles:


All Articles