All geek questions in one place

How to extract source html from Scrapy selector?

I am extracting js data using response.xpath ('// *') re_first () and then converting it to my own python data. The problem is that the extract / re methods do not seem to make it possible not to specify html ie

original html:

{my_fields:['O'Connor Park'], }

extract output:

{my_fields:['O'Connor Park'], }

including this output in json will not work.

What is the easiest way?

+4
source share
3 answers

Short answer:

  • Scrapy / Parsel selectors .re()and .re_first()replace HTML objects (except <, &)
  • .extract() .extract_first(), HTML ( JavaScript) Python re

:

JavaScript HTML.

HTML:

<html lang="en">
<body>
<div>
    <script type="text/javascript">
        var i = {a:['O&#39;Connor Park']}
    </script>
</div>
</body>
</html>

scrapy, parsel, Javascript:

>>> import scrapy
>>> t = """<html lang="en">
... <body>
... <div>
...     <script type="text/javascript">
...         var i = {a:['O&#39;Connor Park']}
...     </script>
...     
... </div>
... </body>
... </html>
... """
>>> selector = scrapy.Selector(text=t, type="html")
>>> 
>>> # extracting the <script> element as raw HTML
>>> selector.xpath('//div/script').extract_first()
u'<script type="text/javascript">\n        var i = {a:[\'O&#39;Connor Park\']}\n    </script>'
>>> 
>>> # only getting the text node inside the <script> element
>>> selector.xpath('//div/script/text()').extract_first()
u"\n        var i = {a:['O&#39;Connor Park']}\n    "
>>>

, .re ( .re_first), :

>>> # I'm using a very simple "catch-all" regex
>>> # you are probably using a regex to extract
>>> # that specific "O'Connor Park" string
>>> selector.xpath('//div/script/text()').re_first('.+')
u"        var i = {a:['O'Connor Park']}"
>>> 
>>> # .re() on the element itself, one needs to handle newlines
>>> selector.xpath('//div/script').re_first('.+')
u'<script type="text/javascript">'    # only first line extracted
>>> import re
>>> selector.xpath('//div/script').re_first(re.compile('.+', re.DOTALL))
u'<script type="text/javascript">\n        var i = {a:[\'O\'Connor Park\']}\n    </script>'
>>>

HTML &#39; apostrophe. w3lib.html.replace_entities() .re/re_first (. parsel , extract_regex), extract() extract_first()

+8

, Selector class 'extract, :

from lxml import etree
etree.tostring(selector._root)
+1

Starting with version 1.2.0 (2017-05-17), you can pass replace_entities=Falseboth reand re_firstto avoid the default behavior.

0
source

Source: https://habr.com/ru/post/1625035/

More articles:


All Articles