Add data to HDF5 file using Pandas, Python

Question

I have big pandas DataFrames with financial data. I have no problem adding and concatenating extra columns and DataFrames to my .h5 file.

The financial data is updated every minute, I need to add a row of data to all my existing tables inside my .h5 file every minute.

Here is what I have tried so far, but no matter what I do, it overwrites the .h5 file and doesn’t just add the data.

HDFStore:

#we open the hdf5 file
save_hdf = HDFStore('test.h5') 

ohlcv_candle.to_hdf('test.h5')

#we give the dataframe a key value
#format=table so we can append data
save_hdf.put('name_of_frame',ohlcv_candle, format='table',  data_columns=True)

#we print our dataframe by calling the hdf file with the key
#just doing this as a test
print(save_hdf['name_of_frame'])    

---

In another way, I tried this, to_hdf:

#format=t so we can append data , mode=r+ to specify the file exists and
#we want to append to it
tohlcv_candle.to_hdf('test.h5',key='this_is_a_key', mode='r+', format='t')

#again just printing to check if it worked 
print(pd.read_hdf('test.h5', key='this_is_a_key'))

---

Here's what one of the DataFrames looks like after reading hhdf:

           time     open     high      low    close     volume           PP  
0    1505305260  3137.89  3147.15  3121.17  3146.94   6.205397  3138.420000   
1    1505305320  3146.86  3159.99  3130.00  3159.88   8.935962  3149.956667   
2    1505305380  3159.96  3160.00  3159.37  3159.66   4.524017  3159.676667   
3    1505305440  3159.66  3175.51  3151.08  3175.51   8.717610  3167.366667   
4    1505305500  3175.25  3175.53  3170.44  3175.53   3.187453  3173.833333  

, ( ), , ​​ 5 ... 6 7.. , , . , .

P.S.

Answer 1

pandas.HDFStore.put() append ( False), Pandas .

:

store = pd.HDFStore('test.h5')

store.append('name_of_frame', ohlcv_candle, format='t',  data_columns=True)

store.put(..., append=True), :

store.put('name_of_frame', ohlcv_candle, format='t', append=True, data_columns=True)

: format='t' table (format='t' - format='table').

Answer 2

tohlcv_candle.to_hdf('test.h5',key='this_is_a_key', append=True, mode='r+', format='t')

append=True , append=True , , .

False 'this_is_a_key' .

mode= , .

, mode='a', append=False , , .

, , . , .

: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_hdf.html.

: hdf5 - . , .