Django / Comet (Push): Least of all evil?

Question

I have read all the questions and answers I can find regarding Django and HTTP Push. However, none of them offers a clear, concise, end-to-end solution on how to fulfill the basic “hello world” of the so-called “comet”.

First question (1): To what extent is the problem that HTTP is simply not (at least so far) for this? Are all potential solutions hacked substantially?

2) What is the best solution currently?

• flew around?
• Some other Twisted based solutions?
• Tornado?
• node.js?
• XMPP w / BOSH?

Another solution?

3) How does the nginx push module work in this discussion?

4) Which of these solutions requires replacing the typical deployment model mod_wsgi / nginx (or apache)? Why do they need it? Is this a favorable transition anyway?

5) How important are the benefits of using a solution that is already in Python?

The performance by Alex Gynor from PyCon 2010 that I just watched on blip.tv is awesome and informative, but not the worst for the current state of HTTP Push in Django. One thing that he said gave me some confidence is this: Orbited does a good job of abstracting and modeling the concept of network sockets. That way, when the WebSockets actually land, we will be in a good place to transition.

6) How are HTML5 websites different from existing solutions? Is Gaynor’s assessment of the ease of transition from Orbital accurate?

Answer 1

I cannot believe that more than six years have passed since I asked this question.

Async with Django (and related network traffic like websockets) was for many of us in the community. I took these last few years, among other things, scratched this itch.

hendrix

hendrix is ​​a WSGI / ASGI conatiner that runs on Twisted. This is a project mainly focused on 5 enthusiasts, with help and funding from some visionary organizations. It is in production today by dozens, but not hundreds of companies.

I will leave this to you to read the documentation to understand why this is the best solution to this problem, but with a few quick shortcuts:

• It is based on Twisted, does not require knowledge or use of Twisted internal components, but leaves them available
• It “just works” in the sense that you don’t need any special server or process configuration to make asynchronous and socket traffic from your Django application (or Pyramid or Flask)
• Most likely, it will be compatible with ASGI, the Django channel standard, and has several significant ways the first ASGI container
• It comes with simple APIs that support the flow of your browsing logic and are easily accessible by unit test.

Please see this talk I gave at Django-NYC (at Buzzfeed offices) for more information on why I think this is the best answer to this question.

Answer 2

I would look at evserver (http://code.google.com/p/evserver/) if you need a comet.

It "supports a little-known extension to the asynchronous WSGI" and builds around libevent. Works like a charm and supports django. The actual handler code is a little ugly, but it scales well, as it is really asynchronous.

I used evserver, and now I'm going to the cyclone (tornado on a twisted), because I need a little more than evserver. I need a real bidirectional io (think socket.io (http://socket.io/)), and although evserver could support it, I thought it was easier to override tornado socket.io in a cyclone (I chose a cyclone instead of a tornado, since the cyclone is built on twisted, which allows you to use more transports that are not implemented in twisted (ic zeromq)). Socket.io supports web connectors, comet-style polling, and much more flash-based firewalls. I think that in most practical situations, websockets + flash-based websockets are enough to support 99% (according to adobe flash penetration about 99% (http://www.adobe.com/products/player_census/flashplayer/version_penetration.html )) visitors to websites (only people who do not use flash memory should lean back on one of the sockets. Its backup (less performance and resource))

Remember that websites are not http vehicles , so they put them in http-based proxies (like haproxy in http mode), breaking the connection. Better serve them on an alternative ip or port so that you can proxy the server in tcp mode (for example, haproxy in tcp mode).

To answer your questions: (1) If you don’t need bidirectional multithreading solutions, they are good enough (all they do is keep the connection open). Everything happens if you need your connection to be operational, or you need to be able to send and receive data. In the latter case, socket.io helps. However, for this scenario, websites are created and with flash support available for most websites (via socket.io or offline, however socket.io has the added benefit of backup transport for those people who do not want to install flash)

(2) if you just need to click, evserver is your best bet. It uses the same client-side javascripts as in orbit. Let's look at socket.io again (for this, you also need a supporting server, the only python available is tornado.)

(3) This is just one other server implementation. If I read it correctly, click only. pushing data to the client is done by installing http equest from your application to the nginx server. (nginx then makes sure they reach the client). If you are interested in this, look at mongrel2 (http://mongrel2.org/home), it has not only handlers for longpolling, but also for websockets. (Instead of making an http request to mongrel, this time you use zeromq handlers to receive data on your mongrel server) (note the lack of developer enthusiasm for websites and flash sites. Especially considering that websocket protocol tends to evolve, at some point you may need to transcode mongrel2 web wallpaper support with websockets support)

(4) All solutions except evserver replace wsgi with something else. Although most servers also have some wsgi support on this "something else". No matter which solution you choose, be careful that one intensive processor or another io lock request does not block the server. (you need either multiple instances or threads).

(5) Not very important. All decisions depend on some user agents to push (and, if applicable, receive) data to the client. All the solutions mentioned allow you to write these handlers in python. If you want to use a completely different structure (node.js), then you need to weigh the ease of node.js (it is considered simple, but it is also quite experimental, and I found very few libraries that are really stable) from the convenience of using your existing code base and available libraries (for example, if your application needs a blog, you will find many django diaries that you could connect, but not for node.js) Also, don’t look at yourself blindly about performance characteristics. if you don't plan on doing dumb predefined data (which is all tests) for the client, you will find that actual data processing adds a lot more overhead than even the worst asynchronous implementation. (But you still want to use a server based on an asynchronous kernel, if you plan to have many concurrent clients, streaming is simply not designed to save thousands of connections)

(6) websockets offer bidirectional communication, long polling / comet only pushes data, but does not accept records. (Socket.io simulates this bi-directional support using two HTTP requests, one for longpoll, one for sending data. It tracks their interdependence with an identifier (session), which is part of the request string of both requests). flash-based networks are similar to real web ports (the difference is that they are implemented in swf and not in your browser). Also, the websockets protocol does not comply with the http protocol; longpolling / comet stuff (technically, the websocket client sends an update request to the websocket server, the updated protocol is no longer http)

Answer 3

There is support for WebSockets with django-websocket , but, unfortunately, there are serious problems for its operation; here is a quote from this page:

Disclaimer (what you should know when using django-websocket)

DISCLAIMER BIG FAT - at the moment it is technically NOT possible to use the WSGI website in any way. This is a known issue, but it cannot be handled in a clean way due to some design decision that was made at the time the WSGI stage was written. At this time, things like Websockets, etc., did not exist and were not predictable.

...

But not only WSGI is a limiting factor. Django itself was designed around a simple response script request without using Websockets. It also means that providing standard compliance for a websocket implementation is not possible for django right now. However, this somehow does not work so beautifully. Therefore, keep in mind that tcp sockets can be tortured when using django-websocket.

So, at the moment, WSGI: no; Django: hardly anyone will go, even with django-websockets; see author comment original announcement :

I can’t say it looks good. You make long-lived connections in such a way as to require threads. django-websocket requires a streaming installation and will not work if you have processes (because you just have too many processes), but the streams will not scale for a large number of connections at the same time, so its just a false safety. You need an asynchronous platform for durable things, and I do this by running my application in Django and my comet and websocket in Node.js

Personally, if you are trying to use WebSockets (which I expect next year), I would try a combination of Twisted and Cyclone . They are designed to work with WebSockets and scale well. If you write your code correctly to remove unnecessary dependencies on Django, you can use most of your code on a Twisted-based system. This is a very great advantage over using Node.js or a comet or any system in another language. You can also do a simple push

Finally, you can also just solve it too complicated and use an external service to provide push support. Then the question arises of sending a simple JSON request to their servers, and not worry about how to establish a connection and how concurrency will work, and the like. Of course, you will need to pay for this (although it may currently be free in beta), but you do not need to worry about implementation details; You won’t have the full power of WebSockets, but just click the support button.

Answer 4

Re question # 2, I was recently given a tour of the insides of a Django application that uses Comet heavily, and Orbited is the solution they chose.