Is hdf5 suitable for real-time measurements

Question

I would like to know if hdf5 is suitable for real-time data logging or not?

More precisely: I am working on a project in which we want constantly (sampling frequency in the range from 30 to 400 Hz), mixes quite a lot of data (several hours) of various nature (telemetry, signals, video).

Data should be recorded in real time (or with a slight delay), so that we do not lose them in case of a possible failure.

Our first prototype is based on sqlite3, however, we believe that some restrictions may increase due to prolonged use: speed, one database == one file and difficulties accessing the database from multiple threads (blocking exception when reading and writing to same time).

So, I am considering using hdf5 as a back-end for storing data on disk (and numpy / pytable for internal representation). Do you think it is possible to regularly update hdf5 file from such python binding?

Answer 1

HDF5 Packet Tables Designed for real-time measurements — however, you'd better use packets of a fixed data size to a regular old posix file and convert later. This is due to the fact that HDF5 is currently not very reliable and does not provide various guarantees using a low-level IO input code - this low-level code is actually quite simple to use. At some point, when the data you are working with is quite complex, HDF5 should appear, but be aware of the low level IO file, it is heavy and cannot be multithreaded with reasonable determinism / performance due to its global use of the mutex. In addition, if the system crashes, the resulting HDF5 file is garbage / unrecoverable - it will be fixed one day, but financing for this HDF group will require funding and do it in the next decade.

My own policy is to use package log files whenever possible. Then immediately convert the result to HDF5 after these files are recorded for long-term use + compression + use by other tools / programs. These recorders often process the HDF5 file explaining the binary structure during recording, so I can just read this file later to understand that the structures are in the log file files and transfer it to a real HDF file after loading the packets in memory.

With all this said and done, browse the api package table from boeing . It also has a C ++ black sheep binding in the hl C ++ library that comes with hdf5, although I had to fix it for my purposes.