All geek questions in one place

Unable to parse Google Play app rating data

The Google Play ecosystem allows you to retrieve ranking data from a bucket, i.e., cloud storage. Although I can successfully download CSV from the Play Developer console and process it, the file is in utf-16 encoding. Here are the first 180 bytes:

'255,254,80,0,97,0,99,0,107,0,97,0,103,0,101,0,32,0,78,0,97,0,109,0,101,0,44,0,65,0,112,0,112,0,32,0,86,0,101,0,114,0,115,0,105,0,111,0,110,0,32,0,67,0,111,0,100,0,101,0,44,0,82,0,101,0,118,0,105,0,101,0,119,0,101,0,114,0,32,0,76,0,97,0,110,0,103,0,117,0,97,0,103,0,101,0,44,0,68,0,101,0,118,0,105,0,99,0,101,0,44,0,82,0,101,0,118,0,105,0,101,0,119,0,32,0,83,0,117,0,98,0,109,0,105,0,116,0,32,0,68,0,97,0,116,0,101,0,32,0,97,0,110,0,100,0,32,0,84,0,105,0,109,0,101,0,44,0,82,0,101,0,118,0,105,0,101,0,119,0'

or decoded:

 u'Package Name,App Version Code,Reviewer Language,Device,Review Submit Date and Time,Review Submit Millis Since Epoch,Review Last Update Date and Time,Review Last Update Millis Since'

However, when I try to access ratings through cloud storage:

 try: stats = cloudstorage.listbucket(folder_path) except Exception as e: raise files = [stat for stat in stats] newest = max(files, key=lambda f: f.st_ctime) newest_ratings_file = cloudstorage.open(newest.filename) s = newest_ratings_file.read() newest_ratings_file.close()

I get the following sequence of bytes (the first is 180 here):

 31,139,8,0,0,0,0,0,0,0,228,189,201,147,44,219,150,222,181,75,74,53,85,42,149,132,74,133,138,82,53,231,189,42,189,71,137,155,85,209,55,247,221,115,239,203,38,154,236,251,150,65,146,125,223,247,73,47,76,48,96,0,162,4,234,104,198,48,131,17,19,16,102,96,134,193,95,112,7,175,94,25,67,13,48,195,140,25,3,224,251,126,219,35,79,158,200,136,188,55,162,210,12,51,220,210,118,100,132,135,251,246,237,238,43,214,222,107,173,111,125,235,255,249,191,103,195,102,216,14,199,122,221,15,187,225,67,152,214,187,83,189,251,34,12,133,11,253,125,8,203,250,116,21,174,195,97,56,15,103,250,60,162,255,59,236,49,175,215,59,109,223,13,247,236,243,33,76,234,232

This is not like any known encoding. Attempting to decode using "utf-8", "utf-16", "utf-16-be", "utf-16-le" leads to decoding errors, for example:

 UnicodeDecodeError: 'utf16' codec can't decode bytes in position 170-171: illegal encoding

CharDetect ( https://pypi.python.org/pypi/chardet ) didn't help either :(

I don’t really understand where to go next. Guys / Girls, any suggestions on how to solve this? Muchas gracias!

+5
source share
1 answer

You can check the contents of Content-Type and Content-Encoding using the cloud storage browser or gsutil ls -L <file> to get an idea of ​​how to decode it.

In this case ( Content-Encoding: gzip Content-Type:text/csv; charset=utf-16le ), the file must be unpacked, then decoded.

+2
source

Source: https://habr.com/ru/post/1232946/

More articles:


All Articles