Simple version control systems or version file system or version database

Question

I am looking for a simple version control system for a large number of records or files (~ 50 million, ~ 100GB unpacked, ~ 20 MB packed). Files contain only a few kilobytes and have unique identifiers, so I do not mind whether they are stored in a flat structure (table, directory ...) or not. On average, each entry changes once a month, but most changes have a difference less than that of Kilobyte, so it is easy to compress. However, a naive database with one record for each version will grow too fast. I need the following operations:

• Basic CRUD operations: create, read, update, delete
• quick list of recent changes
• quick list of recent changes to a specific record
• request changes for a certain period of time
• request changes for this user (each edit is associated with some user ID and may have a commit message as a comment)
• for write operations, there must be a commit latch to check and reject unformatted records.

In short, I'm looking for Wiki-like software for simple entries or files.

I was thinking about possible solutions:

• Put files in version control system . This gives me the ability to replicate and the many access tools available, so this is my preferred solution. But the amount of data is too large for distributed systems such as git. Has anyone successfully used Subversion for a similar task?

• Implement custom versioning in a database or file system. I would only need to store compressed records and differences, it would be more work and learn something. That would be my preferred solution if it were just for fun.

• Use the file system versions . This will make configuration, replication, and access difficult. I will probably need to implement my own access API over the file system.

• Use the version database system . Can you suggest some of them?

• Use any other existing data store with version (MediaWiki ?, Amazon Cloud Drive?, ...)

Obviously there are many fixes. What patterns have others used successfully to produce the same or large amounts of data?

Answer 1

If you don't mind having a raw copy of every file on your client (which, I think, is okay if you are considering svn), then git is probably a pretty good solution to your problem. The base repository repository will use binary differences between files as well as between versions, so you should have close to optimal compression.

With an open repo and some scripts, you can even get away with the lack of a current revision: objects are accessible from the command line, and you can create new commits without checking.