Updating large DataFrames not on disk.

Question

I study all the inputs and outputs of Pandas, manipulating the large csv files received online, the files are time series of financial data. I still figured out how to use HDFStore to store and manage them, but I was wondering if there is an easier way to update files without reloading the entire source file?

I ask because I work with files of size 12 ~ 300 + MB, which update every 15 minutes. Although I do not need the update to be continuous, it would just not download what I already have.

Answer 1

The Blaze library from Continuum should help you. You can find an introduction here .