Validation error with np.load after numpy.savez

I have 5 arrays of numpy a,b,c,d and e , which are defined as:

 array([1, 2, 3, 4, 5, 6, 7, 8, 9]) 

I save these arrays like this:

 np.savez_compressed('tmp/test',a=a,b=b,c=c,d=d,e=e) 

As a result, the test.npz file is test.npz .

However, I am having problems loading data into (below example here ):

 >>> f=np.load('tmp/test.npz') >>> f.files ['a', 'c', 'b', 'e', 'd'] >>> f['a'] 

Gives a large error string ending in:

 File "C:\Python27\lib\compiler\transformer.py", line 754, in atom_lbrace return self.com_dictorsetmaker(nodelist[1]) File "C:\Python27\lib\compiler\transformer.py", line 1214, in com_dictorsetmaker assert nodelist[0] == symbol.dictorsetmaker AssertionError 

I considered using pickle instead. However, this results in files being four times the size of .npz files, so I would like to use savez or savez_compressed .

Does anyone know what I'm doing wrong, or suggestions for alternative approaches?

Here is a script that will result in an error:

 def saver(): import numpy as np a= np.arange(1,10) b=a c=a d=a e=a np.savez_compressed('tmp/test',a=a,b=b,c=c,d=d,e=e) f=np.load('tmp/test.npz') print f.files print f['a'] 

Here is the full trace:

 Traceback (most recent call last): File "<pyshell#16>", line 1, in <module> saver.saver() File "C:\Python27\saver.py", line 14, in saver print f['a'] File "C:\Python27\lib\site-packages\numpy\lib\npyio.py", line 241, in __getitem__ return format.read_array(value) File "C:\Python27\lib\site-packages\numpy\lib\format.py", line 440, in read_array shape, fortran_order, dtype = read_array_header_1_0(fp) File "C:\Python27\lib\site-packages\numpy\lib\format.py", line 336, in read_array_header_1_0 d = safe_eval(header) File "C:\Python27\lib\site-packages\numpy\lib\utils.py", line 1156, in safe_eval ast = compiler.parse(source, mode="eval") File "C:\Python27\lib\compiler\transformer.py", line 53, in parse return Transformer().parseexpr(buf) File "C:\Python27\lib\compiler\transformer.py", line 132, in parseexpr return self.transform(parser.expr(text)) File "C:\Python27\lib\compiler\transformer.py", line 124, in transform return self.compile_node(tree) File "C:\Python27\lib\compiler\transformer.py", line 159, in compile_node return self.eval_input(node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 194, in eval_input return Expression(self.com_node(nodelist[0])) File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node return self._dispatch[node[0]](node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 578, in testlist return self.com_binary(Tuple, nodelist) File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary return self.lookup_node(n)(n[1:]) File "C:\Python27\lib\compiler\transformer.py", line 596, in test then = self.com_node(nodelist[0]) File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node return self._dispatch[node[0]](node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 610, in or_test return self.com_binary(Or, nodelist) File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary return self.lookup_node(n)(n[1:]) File "C:\Python27\lib\compiler\transformer.py", line 615, in and_test return self.com_binary(And, nodelist) File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary return self.lookup_node(n)(n[1:]) File "C:\Python27\lib\compiler\transformer.py", line 619, in not_test result = self.com_node(nodelist[-1]) File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node return self._dispatch[node[0]](node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 626, in comparison node = self.com_node(nodelist[0]) File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node return self._dispatch[node[0]](node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 659, in expr return self.com_binary(Bitor, nodelist) File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary return self.lookup_node(n)(n[1:]) File "C:\Python27\lib\compiler\transformer.py", line 663, in xor_expr return self.com_binary(Bitxor, nodelist) File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary return self.lookup_node(n)(n[1:]) File "C:\Python27\lib\compiler\transformer.py", line 667, in and_expr return self.com_binary(Bitand, nodelist) File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary return self.lookup_node(n)(n[1:]) File "C:\Python27\lib\compiler\transformer.py", line 671, in shift_expr node = self.com_node(nodelist[0]) File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node return self._dispatch[node[0]](node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 683, in arith_expr node = self.com_node(nodelist[0]) File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node return self._dispatch[node[0]](node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 695, in term node = self.com_node(nodelist[0]) File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node return self._dispatch[node[0]](node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 715, in factor node = self.lookup_node(nodelist[-1])(nodelist[-1][1:]) File "C:\Python27\lib\compiler\transformer.py", line 727, in power node = self.com_node(nodelist[0]) File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node return self._dispatch[node[0]](node[1:]) File "C:\Python27\lib\compiler\transformer.py", line 739, in atom return self._atom_dispatch[nodelist[0][0]](nodelist) File "C:\Python27\lib\compiler\transformer.py", line 754, in atom_lbrace return self.com_dictorsetmaker(nodelist[1]) File "C:\Python27\lib\compiler\transformer.py", line 1214, in com_dictorsetmaker assert nodelist[0] == symbol.dictorsetmaker AssertionError 
+4
source share
2 answers

Unable to reproduce your problem on either Linux or Mac (Python 2.7, numpy 1.6.1 / 1.7.1)

But I noticed that you are using a relative path to save the tmp/test.npz . Is it intentional? In my memory, the latest versions of Windows have a special relation to new files that they try to create in some directories (for example, "/ Program Files /") - it repels them, but still says that they exist in some cases. This is not like the case, but can you try the absolute path for the file you save?

BTW: as an alternative to the ZIP archive (which is created by savez savez_compressed ), you can try pickle with 'LZMAFile' as a file object. It gives a very good compression speed (but it can be slow and requires more memory and time when compressing / saving a file);

It is used like any other shell of a file object, something like this (for loading pickled data):

 from lzma import LZMAFile import cPickle as pickle if fileName.endswith('.xz'): dataFile = LZMAFile(fileName,'r') else: dataFile = file(fileName, 'ro') data = pickle.load(dataFile) 
+1
source

I have the same problem (AssertionError) when using numpy 1.7.1 / 1.8.0 with python 2.7.6 on both MAC OS and Windows. But the problem was automatically fixed after switching to Linux using python 2.7.5. Then I reinstall python 2.7.5 on MACOS and Windows, and the whole problem went away. Mostly the problem is python instead of numpy, as the compiler sends alert.So, maybe the version matters.

But although npy is a numpy serializable type, I don't think the file is small enough even with savez_compressed for a large matrix.

I hope your problem will be the same as mine.

0
source

Source: https://habr.com/ru/post/1486829/


All Articles