Optimized bulk (piece) Loading objects into indexed DB

Question

I want to add objects to some table in IndexedDB in one transaction:

_that.bulkSet = function(data, key) { var transaction = _db.transaction([_tblName], "readwrite"), store = transaction.objectStore(_tblName), ii = 0; _bulkKWVals.push(data); _bulkKWKeys.push(key); if (_bulkKWVals.length == 3000) { insertNext(); } function insertNext() { if (ii < _bulkKWVals.length) { store.add(_bulkKWVals[ii], _bulkKWKeys[ii]).onsuccess = insertNext; ++ii; } else { console.log(_bulkKWVals.length); } } }; 

It seems to work fine, but this is not a very optimized way to do this, especially if the number of objects is very large (~ 50.000-500.000). How can I optimize it? Ideally, I want to add the first 3000, and then remove it from the array, and then add another 3000, namely in pieces. Any ideas?

Answer 1

Inserting this number of rows in a row does not produce good performance.

I am IndexedDB dev and have experience working with IndexedDB in real time on the scale you are talking about (writing hundreds of thousands of lines in succession). This is not too sweet.

In my opinion, IDB is not suitable for use when a large amount of data needs to be written sequentially. If I were an IndexedDB application architect who needed a lot of data, I would define a way to seed it gradually over time.

The problem is the recording, and the problem that I see is that the slow recording, combined with the intensity of their intensive I / O, worsens over time. (Readers are always quickly covered in the IDB, for what it's worth.)

To get started, you get the savings from reusing transactions. Because of this, your first instinct may be trying to cut everything into one transaction. But from what I found in Chrome, for example, is that the browser does not seem to like long recordings, perhaps because of some mechanism designed to throttle the wrong tabs.

I'm not sure what performance you see, but average numbers can trick you depending on the size of your test. The limit is faster - bandwidth, but if you are trying to insert large amounts of data, pay attention to records over time.

It turns out that I am working on a demo with several hundred thousand lines at my disposal and have statistics. When my visualization is turned off by running a blank line in IDB, this is what I see now in Chrome 32 in the same object store with a single non-ideal index with automatically increasing the primary key.

A much, much smaller data set of 27 thousand rows, I saw 60-70 records / second:
* ~ 30 seconds: an average of 921 records per second (there is always a large burst of inserts at the beginning), 62 / seconds at the time of selection * ~ 60 seconds: 389 / second average (steady decline, starting with overfulfillment of the initial burst effect) 71 / seconds in the moment


* ~ 1: 30: 258 / second, 67 seconds per second
* ~ 2: 00 (~ 1/3 done): 188 / seconds on average, 66 seconds at a time

Some examples with a much smaller dataset show much better performance, but similar performance. Also much larger datasets - the effects are greatly exaggerated, and I saw only 1 record per second when leaving for several hours.