Pandas _metadata DataFrame save data errors

Question

I finally figured out how to use _metadata from a DataFrame, everything works, except that I cannot save it, for example, for hdf5 or json. I know this works because I copy the frame and _metadata attributes by copying the non_metadata attributes, no.

Example

df = pandas.DataFrame #make up a frame to your liking pandas.DataFrame._metadata = ["testmeta"] df.testmeta = "testmetaval" df.badmeta = "badmetaval" newframe = df.copy() newframe.testmeta -->outputs "testmetaval" newframe.badmeta ---> raises attribute error #json test df.to_json(Path) revivedjsonframe = pandas.io.json.read_json(Path) revivedjsonframe.testmeta ---->raises Attribute Error #hdf5 test revivedhdf5frame.testmeta ---> returns None 

this person https://stackoverflow.com/a/4126146/2126226 says that this worked for him, but I'm new to this site (and pandas) and cannot post to this stream or ask him directly.

Answer 1

_metadata preceded by an underscore, which means that it is not part of the public API. It is not intended for user code - we can search it without any problems in any future version of pandas.

I would highly recommend using this "feature". At the moment, the best option for saving metadata using a DataFrame is probably to write your own wrapper class and do the portability yourself.

Answer 2

This is my code that works using python 3.3.3.2 64-bit

 In [69]: df = pd.DataFrame() #make up a frame to your liking pd.DataFrame._metadata = ["testmeta"] print(pd.DataFrame._metadata) df.testmeta = "testmetaval" df.badmeta = "badmetaval" newframe = df.copy() print(newframe.testmeta) print("newframe", newframe.badmeta) df.to_json(r'c:\data\test.json') read_json = pd.read_json(r'c:\data\test.json') read_json.testmeta print(pd.version.version) print(np.version.full_version) Out[69]: ['testmeta'] testmetaval newframe badmetaval 0.15.2 1.9.1 

JSON content as df:

 In [70]: read_json Out[70]: Empty DataFrame Columns: [] Index: [] In [71]: read_json.info() <class 'pandas.core.frame.DataFrame'> Float64Index: 0 entries Empty DataFrame In [72]: read_json.testmeta Out[72]: 'testmetaval' 

It is strange that the json that is written is just empty parentheses:

{}

which indicates that metadata is actually being distributed across the statement line: pd.DataFrame._metadata = ["testmeta"]

It seems to still work if you overwrite the 2nd atrtibute metadata:

 In [75]: df.testmeta = 'foo' df2 = pd.DataFrame() df2.testmeta = 'bar' read_json = pd.read_json(r'c:\data\test.json') print(read_json.testmeta) print(df2.testmeta) testmetaval bar