I have several (25k) .csv files that I am trying to add to an HDFStore file. They all have the same headings. I use the code below, but for some reason, whenever I run it, the data framework is not added with all the files, but rather is only the last file in the list.
filenames = []
dtypes= {dict of datatypes}
store = pd.HDFStore('store.h5')
store.put('df', pd.read_csv(filenames[0],dtype=dtypes,parse_dates=
["date"]))
for f in filenames:
try:
temp_csv = pd.DataFrame()
temp_csv = pd.read_csv(f,dtype=dtypes,parse_dates=["trade_date"])
store.append('df', temp_csv)
except:
pass
I tried to use a subset of the list of file names, but always get the last entry. For some reason, the loop does not add my file, but overwrites it every time. Any advice would be appreciated, as it made me work. (python 3, windows)
source
share