Saving Pandas objects with regular Python objects in HDF5

Question

Pandas has a nice interface that makes it easy to store things like Dataframes and Series in HDF5:

random_matrix = np.random.random_integers(0,10, m_size) my_dataframe = pd.DataFrame(random_matrix) store = pd.HDFStore('some_file.h5',complevel=9, complib='bzip2') store['my_dataframe'] = my_dataframe store.close() 

But if I try to save some other regular Python objects in a single file, it complains:

 my_dictionary = dict() my_dictionary['a'] = 2 # <--- ERROR my_dictionary['b'] = [2,3,4] store['my_dictionary'] = my_dictionary store.close() 

from

 TypeError: cannot properly create the storer for: [_TYPE_MAP] [group->/par ameters (Group) u'',value-><type 'dict'>,table->None,append->False,kwargs- >{}] 

How can I store regular Python data structures in one HDF5 where I store other Pandas objects?

Answer 1

Here is a cookbook example: http://pandas.pydata.org/pandas-docs/stable/cookbook.html#hdfstore

You can store arbitrary objects as node attributes. I believe that there is a limit of 64 KB (I think its general attribute data for this node). Pickled objects

 In [1]: df = DataFrame(np.random.randn(8,3)) In [2]: store = HDFStore('test.h5') In [3]: store['df'] = df # you can store an arbitrary python object via pickle In [4]: store.get_storer('df').attrs.my_attribute = dict(A = 10) In [5]: store.get_storer('df').attrs.my_attribute {'A': 10}