UnicodeEncodeError when selecting URLs

Question

I am using urlfetch to get the url. When I try to send it to the html2text function (removes all HTML tags), I get the following message:

UnicodeEncodeError: 'charmap' codec can't encode characters in position  ... character maps to <undefined>

I am trying to handle the encoding ('UTF-8', 'ignore') in a string, but I keep getting this error.

Any ideas?

Thank,

Joel

---

The code:

result = urlfetch.fetch(url="http://www.google.com")
html2text(result.content.encode('utf-8', 'ignore'))

And the error message:

File "C:\Python26\lib\encodings\cp1252.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 159-165: character maps to <undefined>

Answer 1

You need to decode the data that you selected first! What codec? Depends on the website you receive.

When you have Unicode and try to encode it with some_unicode.encode('utf-8', 'ignore'), I cannot figure out how this could cause an error.

Good thing you need to do:

result = fetch('http://google.com') 
content_type = result.headers['Content-Type'] # figure out what you just fetched
ctype, charset = content_type.split(';')
encoding = charset[len(' charset='):] # get the encoding
print encoding # ie ISO-8859-1
utext = result.content.decode(encoding) # now you have unicode
text = utext.encode('utf8', 'ignore') # encode to uft8

, .