How to migrate large objects using postMessage webworker?

Question

I read that portable objects can be transferred very quickly using postmessage web worker. According to these, the transferred objects are either arraybuffer or messageport.

The question is how to convert, for example, an arbitrary object with a large size (30 mb) to a portable object and pass it as an argument to postmessage. From what I understand, I can convert my array to a json string and then convert the json string to the original byte data and store this inside the array object. However, this seems to be the goal of fast transmission.

can someone enlighten me to pass an object as a portable object or if it is possible?

Thanks in advance!

Answer 1

This misconception is often repeated here. You imagine that you can write some quick javascript code to convert your large object into a portable one. But really, any conversion code you write hits the target , as you said. And the more complex the data, the more you lose speed.

Typically, objects (when not transmitted) are converted based on a structured cloning algorithm (which uses the format defined during implementation and is optimal). Any javascript code that you write is more likely to be slower than a structured clone if you achieve the same goal - transferring data as binary files.

The purpose of portable objects is to allow the transfer of binary data, such as images (from the canvas), audio or video. These children of data can be transferred without processing using a structured cloning algorithm, so for some reason a portable interface was added. And the effect is small even for them - see the answer about the transmission speed .

As a final note, I wrote a prototype library that converts javascript objects to and from ArrayBuffer. This is slower, especially for JSON, like data. Benefits (and benefits of any similar code you write):

• You can define custom object conversions
• You can use inheritance (e.g. send your own types like Foo )

Code to transfer a JSON object as

If your data is like JSON, just stick with a structured clone and don't carry over. If you do not trust me, check it with this code. You will see it slower than usual. postMessage .

 var object = {dd:"ddd", sub:{xx:"dd"}, num:666}; var string = JSON.stringify(object); var uint8_array = new TextEncoder(document.characterSet.toLowerCase()).encode(string); var array_buffer = uint8_array.buffer; // now transfer array buffer worker.postMessage(array_buffer, [array_buffer]) 

The opposite conversion, given that you have an ArrayBuffer :

 // Let me just generate some array buffer for the simulation var array_buffer = new Uint8Array([123,34,100,100,34,58,34,100,100,100,34,44,34,115,117,98,34,58,123,34,120,120,34,58,34,100,100,34,125,44,34,110,117,109,34,58,54,54,54,125]).buffer; // Now to the decoding var decoder = new TextDecoder("utf-8"); var view = new DataView(array_buffer, 0, array_buffer.byteLength); var string = decoder.decode(view); var object = JSON.parse(string); 

Answer 2

Had to find Thomas's answer earlier.

The proof, though not as Thomas suggested.

Version A

Version B

I manually converted the gated json obejct to Uint8Array as follows:

 function stringToUintArray(message) { var encoded = self.btoa(message); var uintArray = Array.prototype.slice.call(encoded).map(ch => ch.charCodeAt(0)); var uarray = new Uint8Array(uintArray); return uarray; } 

and passed it like this from a web worker to the main thread:

 console.time('generate'); var result = generate(params.low, params.high, params.interval, params.size); var uarr = stringToUintArray(JSON.stringify(result)); console.timeEnd('generate'); self.postMessage(uarr.buffer, [uarr.buffer]); 

and in the main thread, I did something like this:

 var uarr = new Uint8Array(e.data); var json = UintArrayToString(uarr); var result = JSON.parse(json);