Data Transfer via WebSocket

Question

since I use WebSocket connections on more regular bases, I was interested in how everything works under the hood. Thus, I have been breaking into endless specifications for a long time, but so far I have not been able to find anything about how to spoof the stream itself.

The WebSocket protocol calls it data frames (which describes a clean data stream, which is why it is also called non-control frames). As far as I understand the specification, there is no defined maximum length and no specified MTU (maximum transmission unit), which, in turn, means that one frame of WebSocket data can contain, according to the specification (!) An infinite amount of data (please correct me if I'm wrong, I'm still involved in this).

After reading this, I instantly set up my small Node WebSocket server. Since I have a strong history of Ajax (also streaming and Comet), my expectations were original: "There must be some kind of interactive mode for reading data while it is being transmitted." But I'm wrong, right?

I started small with 4kb of data.

server

testSocket.emit( 'data', new Array( 4096 ).join( 'X' ) ); 

and, as expected, it will come to the client as one data block

client

 wsInstance.onmessage = function( data ) { console.log( data.length ); // 4095 }; 

therefore, I increased the payload, and I really expected again that at some point the client-side onmessage would fire again, effectively blocking the transfer. But, to my shock, this never happened (node-server, tested on the side of Firefox, chrome and safari on the client side). My biggest payload was 80 MB

 testSocket.emit( 'data', new Array( 1024*1024*80 ).join( 'X' ) ); 

and he still arrived in one big block of data on the client. Of course, this takes some time, even if you have a pretty good connection. Questions here

• Is it possible to cut these streams similar to ReadyState3 XHR mode?
• Is there a size limit for one ws data frame?
• Are these websites that should not transmit such large payloads? (which would make me wonder again why there is no certain maximum size)

I can still look at WebSockets from the wrong point of view, maybe the need to send large amounts of data simply does not exist, and you have to completely or partially split any data before sending?

Answer 1

First, you need to distinguish between the WebSocket protocol and the WebSocket API in browsers.

The WebSocket protocol has a frame size limit of 2 ^ 63 octets, but a WebSocket message can consist of an unlimited number of frames.

The WebSocket API in browsers does not provide a frame-based or streaming-based API, but only a message-based API. The payload of the incoming message is always fully buffered (as part of the WebSocket browser implementation) before providing its JavaScript.

The APIs of other WebSocket implementations may provide access to a frame transmitted via the WebSocket protocol based on frame or streaming. For example, AutobahnPython . You can read more in the examples here https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/websocket/streaming .

Disclosure: I am the original author of Autobahn and work for Tavendo.

Other considerations:

While the JS Websocket Browser API is missing, you can only receive or send full WS messages.

A single (regular) WebSocket connection cannot interleave the payload of multiple messages. That is, if you use large messages, they are delivered in order, and you cannot send small messages between them while the large message is still on the go.

There is the following WebSocket extension (extensions are an integrated mechanism for protocol expansion): WebSocket multiplexing. This allows you to have multiple (logical) WebSocket connections over the same basic TCP connection, which has several advantages.

Please note: you can simultaneously open several WS-connections (at different base TCP-addresses) to one target server from one JS / HTML page.

Also note: you can “break up” yourself at the application level: send your stuff in smaller WS messages to collect yourself.

I agree that in an ideal world you will have the / Frame / Streaming API in your browser and WebSocket multiplexing. This will give all the strength and convenience.

Answer 2

RFC 6455 Section 1.1 :

This is what the WebSocket protocol provides: [...] an alternative to HTTP polling for two-way communication from a web page to a remote server .

As indicated, WebSockets are intended for communication between a web page and a server. Note the difference between a webpage and a web browser. Examples are browser games and chat applications that read a lot of small messages.

If you want to send a lot of MB in one message, I think you are not using WebSockets the way they were intended. If you want to transfer files, do this using the Plain Old Http Request, which is answered by Content-Disposition , so that the browser downloads the file.

So, if you explain why you want to send such large amounts of data, maybe someone can help come up with a more elegant solution than using WebSockets.

In addition, the client or server may refuse too large messages (although it is not explicitly stated how this will be rejected):

RFC 6455 Section 10.4 :

Implementations that have an implementation and / or platform for limiting the frame size or the total message size after reassembling from several frames MUST protect themselves from exceeding these limits. (For example, a malicious endpoint might try to issue its peer memory or set up a denial of service attack by sending either one large frame (for example, 2 ** 60 in size) or sending a long stream of small frames that are part of a fragmented message.) This implementation SHOULD impose a restriction on the frame size and the total size of the message after reassembly of several frames.