Error starting INSERT in MySQL

Question

_mysql_exceptions.Warning: Incorrect string value: '\xE7\xB9\x81\xE9\xAB\x94...' for column 'html' at row 1


def getSource(theurl, moved = 0):
    if moved == 1:
        theurl = urllib2.urlopen(theurl).geturl()
    urlReq = urllib2.Request(theurl)
    urlReq.add_header('User-Agent',random.choice(agents))
    urlResponse = urllib2.urlopen(urlReq)
    htmlSource = urlResponse.read()
    return htmlSource

new_u = Url(source_url = source_url, source_url_short = source_url_short, source_url_hash = source_url_hash, html = htmlSource)
new_u.save()

Why is this happening? I basically load the URL of the page ... and then save it to the database using Django.

This happens sometimes ... and sometimes it works fine.

Edit: looks like I should install the database in UTF-8? What kind of team is this?

Answer 1

Basically you need to ensure the correct encoding of strings. For example. the string you provide for django is not UTF-8 encoded, and therefore some characters cannot be resolved.

Some useful tips on how to find the encoding of the requested page can be found here: urllib2 read in Unicode

Answer 2

, MySQL. , . MySQL Alter, - : MySQL Alter Table.

, , . , , . "utf8" - .

UTF8 django MySQL.