Error in official scripting example?

Question

I tried using an example of using scrapy on a page (an example under the name: returning several requests and elements from one callback)

I just changed the domains to point to a real website:

import scrapy

class MySpider(scrapy.Spider):
    name = 'huffingtonpost'
    allowed_domains = ['huffingtonpost.com/']
    start_urls = [
        'http://www.huffingtonpost.com/politics/',
        'http://www.huffingtonpost.com/entertainment/',
        'http://www.huffingtonpost.com/media/',
    ]

    def parse(self, response):
        for h3 in response.xpath('//h3').extract():
            yield {"title": h3}

        for url in response.xpath('//a/@href').extract():
            yield scrapy.Request(url, callback=self.parse)

But getting ValuErroras published in that sense . Any ideas?

Answer 1

Some extracted links are relative (for example, /news/hillary-clinton/). You must convert it to absolute (http://www.huffingtonpost.com/news/hillary-clinton/

import scrapy

class MySpider(scrapy.Spider):
    name = 'huffingtonpost'
    allowed_domains = ['huffingtonpost.com/']
    start_urls = [
        'http://www.huffingtonpost.com/politics/',
        'http://www.huffingtonpost.com/entertainment/',
        'http://www.huffingtonpost.com/media/',
    ]

    def parse(self, response):
        for h3 in response.xpath('//h3').extract():
            yield {"title": h3}

        for url in response.xpath('//a/@href').extract():
            if url.startswith('/'):
                # transform url into absolute
                url = 'http://www.huffingtonpost.com' + url
            if url.startswith('#'):
                # ignore href starts with #
                continue
            yield scrapy.Request(url, callback=self.parse)