I am new to PyTables and I am considering using it to process data created using agent-based simulation modeling and stored in HDF5. I am working with a 39 MB test file and am experiencing some weirdness. Here's the location of the table:
/example/agt_coords (Table(2000000,)) ''
description := {
"agent": Int32Col(shape=(), dflt=0, pos=0),
"x": Float64Col(shape=(), dflt=0.0, pos=1),
"y": Float64Col(shape=(), dflt=0.0, pos=2)}
byteorder := 'little'
chunkshape := (20000,)
This is how I access it in Python:
from tables import *
>>> h5file = openFile("alternate_hose_test.h5", "a")
h5file.root.example.agt_coords
/example/agt_coords (Table(2000000,)) ''
description := {
"agent": Int32Col(shape=(), dflt=0, pos=0),
"x": Float64Col(shape=(), dflt=0.0, pos=1),
"y": Float64Col(shape=(), dflt=0.0, pos=2)}
byteorder := 'little'
chunkshape := (20000,)
>>> coords = h5file.root.example.agt_coords
Now that things are getting weird.
[x for x in coords[1:100] if x['agent'] == 1]
[(1, 25.0, 78.0), (1, 25.0, 78.0)]
>>> [x for x in coords if x['agent'] == 1]
[(1000000, 25.0, 78.0), (1000000, 25.0, 78.0)]
>>> [x for x in coords.iterrows() if x['agent'] == 1]
[(1000000, 25.0, 78.0), (1000000, 25.0, 78.0)]
>>> [x['agent'] for x in coords[1:100] if x['agent'] == 1]
[1, 1]
>>> [x['agent'] for x in coords if x['agent'] == 1]
[1, 1]
I don’t understand why the values are screwed up when I iterate over the whole table, but not when I take a small subset of the whole set of rows. I am sure that this is a mistake in the way I use the library, so any help in this matter would be greatly appreciated.