Is there an ACID structure to ensure continuity of bulk data, which would also allow some basic search capabilities to be used? I'm not looking for a full-blown DBMS, but rather something fast, easy, and simple. Even just taking care of atomic commits would be great just to avoid reusing this in the event of a power failure.
SQL Server is too slow for this and has too much overhead; SQLite is even slower (with potentially less overhead?).
Basically, I need to store large amounts of temporary data for every second. As normalized data, this will correspond to ~ 10k rows of the table, but as binary data, it can be represented using ~ 200kb. Obviously, writing 200 kb to disk is a piece of cake compared to writing 10k lines to a relational database.
I could just save it in one or more large binary files, and then implement my own indexing to provide quick filtering on certain fields, but the only thing that scares me is non-atomic transactions and read / write lock scripts.
Any recommendations? I use C # btw, so it would be advisable to have something with .NET wrappers.
[Edit] Regarding ACID, I just found this, for example: Managed Shell for Transactional NTFS (although TxF is a feature of Vista and later).
source
share