I am creating a web application in python. Part of this application works with data that can be described as follows:
Symbol Begin Date End Date
AAPL Jan-1-1985 Dec-27-2010
...
The data is somewhat static - they will be updated periodically, that is: new records can be added, and the "End date" field can be updated for all records.
Now, the question is: given the more or less static nature of the data set, what is the best way to store and work with it? "Job" means fetching random strings, hopefully more than a few times per second.
I can do this with an XML file, with SQL DB or SQLite, with a JSON object file and some kind of python object in memory.
What are the pros and cons of different solutions? I will be grateful for the explanations and for the extreme situation (for example, until the XML file up to 10 times / sec becomes the best after SQL DB).
Update: Thanks for all the answers! Just a small update: currently the set is about 3 thousand lines. It can grow to, say, 15 thousand lines per year. Access pattern: updates are regular, once a day, for a full set; therefore, both the addition of rows and the updating of the end date will be performed immediately. Fetching a random string is valid by character, can be performed several times per second.
source
share