Pycurl handle hanging on twitter streaming api

Question

I am using pycurl to connect to the twitter streaming API.

This works well, but sometimes after starting up for several hours, it stops hanging indefinitely without throwing any exceptions. How can I detect / handle a hang in this script?

import pycurl, json

STREAM_URL = "http://stream.twitter.com/1/statuses/filter.json"

USER = "presidentskroob"
PASS = "12345"

def on_receive(data):
  print data

conn = pycurl.Curl()
conn.setopt(pycurl.USERPWD, "%s:%s" % (USER, PASS))
conn.setopt(pycurl.URL, STREAM_URL)
conn.setopt(pycurl.WRITEFUNCTION, on_receive)
conn.perform()

Answer 1

FROM: http://man-wiki.net/index.php/3:curl_easy_setopt

CURLOPT_LOW_SPEED_LIMIT. Skip the long parameter. It contains the transfer rate in bytes per second, so that the transfer is lower for CURLOPT_LOW_SPEED_TIMEseconds, so that the library considers it too slowly and interrupts.

and

CURLOPT_LOW_SPEED_TIME. . , CURLOPT_LOW_SPEED_LIMIT , .

---

:

conn.setopt(pycurl.LOW_SPEED_LIMIT, 1)
conn.setopt(pycurl.LOW_SPEED_TIME, 90)

Answer 2

- , . , , , API Stream Streaming 1/30, 30 . , , 1 /, , ( ) , . , 30- , 30 .

curl -d @filter.txt https://stream.twitter.com/1/statuses/filter.json -uTwitterLogin:TwitterPassword --speed-time 30 --speed-limit 1

: , curl.

Answer 3

:

 conn.setopt(pycurl.CONNECTTIMEOUT, 15) 
 conn.setopt(pycurl.TIMEOUT, 25) 

pycurl.error, .

Answer 4

I have a hunch that this might be due to the "tcp broken pipe" script. That is, another peer at some point closes the connection, but our partner somehow ignores this event. You will need to use some kind of retention for this.

A “correct”, elegant solution to the problem may require some action from the twitter itself. This is a fairly common problem; my friend used a streaming api and ran into the same problem.