The best place to store large amounts of session data

Question

I am creating an application that needs to store and reuse large amounts of data per session.

So, for example, the user selects a large list of list items (say, 2000 or much more) that have a numerical value as their key, then save this choice and go to another page, do something else, and then return to the original page and you need to upload their selection to this page.

What is the fastest and most efficient way to store and reuse this data?

In a text file saved with a session id?

In the temporary database table?

In the session data itself (DB sessions whose size is not a limit) using a serialized string or using gzcompress or gzencode ?

Answer 1

Although I usually always recommend that users store their data in a database rather than in simple files, this is an exception. In general, there is little overhead when storing data in a database compared to files, but the former provides greater access flexibility and eliminates many locking problems. However, if you do not expect your page to become especially slow for users with multiple browsers accessing the same session, then concurrency will not be a big problem, i.e.

Using any database will be slower

(also, if you intend to work with a large cluster of web servers — more than 200 — to use the same session, then yes, a distributed database can outperform the cluster file system in a SAN).

You probably want to think about how often the session will be written. By default, the handler writes data back to disk every time, regardless of whether it has been changed or not - for such a large session, I suggest you use your own session handler, which not only writes serialized session data to a file, but also stores a hash of serialized data - when you read in a session, store the hash in a static variable. In the save handler, create a new hash and compare it with the static variable filled at boot time - just record the session if it has changed. You can continue this by using heuristics to split the session into parts that are frequently updated and parts that change less frequently and then write them to separate files.

Using compression for this will not really help in performance.

Of course, to optimize the OS level settings, there are enough opportunities for optimization, but you do not say what your OS is. Assuming its POSIX and your system are not yet on their knees, your prevalence hits will be:

Data file access delay and data analysis

(the time to read the file is relatively small, and the record should be buffered).

As long as there is enough cache, the file will be read from memory, not from disk, so the wait time will be short.

FROM.

Answer 2

anyway you want. But no matter what you choose, the array will be serialized into a string and stored either in a file (implicitly using sessions) or in a database field. Reading and writing is faster in the file, so this is a search. I see no reason to use a database for this.

For alternative serialization, check out this tool: http://msgpack.sourceforge.net/

Answer 3

A database will work for this. Just associate the session with the visitors table and get a table called visitor_list_items that saves the selected items as strings.

2000 is not a crazy number to retrieve. I mean, geez, if they are going to sit and select 2,000 list items, they can wait one second to load the page! (Are you sure there is no way to break this selection process into steps?)

If it is in db, you can use regular db applications (i.e. it is easier to run reports about which users choose when they come to your site, etc.).