Python: How to read and parse an unfode utf-8 text file?

Question

I am exporting UTF-8 text from Excel and I want to read and parse incoming data using Python. I read all the online information, so I already tried this, for example:

 txtFile = codecs.open( 'halout.txt', 'r', 'utf-8' )
 for line in txtFile:
  print repr( line )

The error I am getting is:

UnicodeDecodeError: codec 'utf8' cannot decode byte 0xff at position 0: unexpected byte code

Looking at the text file in the Hex editor, I also tried the first FFFE values:

txtFile.seek( 2 )

right after "open", but it just causes a different error.

Answer 1

This is a specification.

EDIT, from comments, this seems to be utf-16 bom

codecs.open('foo.txt', 'r', 'utf-16')

must work.

Answer 2

UTF-8; UTF-16LE .

Answer 3

Expanding Jonathan's comment, this code should read the file correctly:

import codecs
txtFile = codecs.open( 'halout.txt', 'r', 'utf-16' )
for line in txtFile:
   print repr( line )

Answer 4

Try to see if there are several blank lines in the excel file (and then the values ​​again), which may cause an unexpected error.