Missing Unicode codecs on kivy / Python on Android?

Question

I have a program running kivy that works fine on Windows but cannot open the file on Android (1.8.0 on both platforms). The strange thing is that the error message indicates an attempt to decode ASCII when I just ask utf-8; also the file does not have a single 0xFF character in it - I checked with the utility od.

The code:

try:
    Logger.info('Mark: opening file ' + repr(filename))
    with codecs.open(filename, 'r', encoding='utf_8') as f:
        lines = [line.rstrip(u'\r\n').lstrip(codecs.BOM) for line in f]
except UnicodeDecodeError as e:
    Logger.info('Mark: utf8 load failed: ' + str(e))
    with codecs.open(filename, 'r', encoding='cp1252') as f:
        lines = [line.rstrip(u'\r\n') for line in f]

Log file:

[INFO              ] Mark: opening file '/mnt/sdcard/My Files/Documents/DVDs.txt'
[INFO              ] Mark: utf8 load failed: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
[INFO              ] Base: Leaving application in progress...
[WARNING           ] stderr: Traceback (most recent call last):
[WARNING           ] stderr:   File "main.py", line 295, in <module>
[WARNING           ] stderr:     app.run()
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/app.py", line 792, in run
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/base.py", line 481, in runTouchApp
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/core/window/window_pygame.py", line 381, in mainloop
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/core/window/window_pygame.py", line 287, in _mainloop
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/base.py", line 321, in idle
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/clock.py", line 422, in tick
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/clock.py", line 537, in _process_events
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/clock.py", line 309, in tick
[WARNING           ] stderr:   File "main.py", line 266, in select_file_part2
[WARNING           ] stderr:     with codecs.open(filename, 'r', encoding='cp1252') as f:
[WARNING           ] stderr:   File "/home/tito/code/python-for-android/build/python-install/lib/python2.7/codecs.py", line 884, in open
[WARNING           ] stderr: LookupError: unknown encoding: cp1252

Edit: I entered the debug code to go through the entire list and found that everything was missing without exception ascii, latin_1and various codecs utf.

Answer 1

try/except . codecs.BOM \xff\xfe, Unicode ascii

>>> import codecs
>>> codecs.BOM
'\xff\xfe'
>>> u'test'.lstrip(codecs.BOM)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)