TweepError Failed to parse JSON payload - random error

I am using the Tweepy package in Python to collect tweets. I track several users and collect their latest tweets. For some users, I get the error "Failed to parse the JSON payload:", for example. "Failed to parse the JSON payload:" Pending "," delimiter "or"} ": row 1 of column 694303 (char 694302)." I took note of the user id and tried to reproduce the error and debug the code. The second time I ran the code for this particular user, I got the results (i.e. Tweets) without any problems. I adjusted my code so that when I get this error, I will try to extract the tweets again. Thus, I can get this error once or twice for the user, but in the second or third attempt, the code returns tweets,as usual, without error. I get similar behavior for other users as well.

My question is: why does this error appear randomly? Nothing else has changed. I searched the Internet but could not find a similar report. A snapshot of my code follows

#initialize a list to hold all the tweepy Tweets
alltweets = []
ntries = 0
#make initial request for most recent tweets (200 is the maximum allowed count)
while True:
    try: #if process fails due to connection problems, retry.
        if beforeid:
            new_tweets = api.user_timeline(user_id = user,count=200, since_id=sinceid, max_id=beforeid)
        else:
            new_tweets = api.user_timeline(user_id = user,count=200, since_id=sinceid)
        break
    except tweepy.error.RateLimitError:
        print "Rate limit error:", sys.exc_info()[0]
        print("Timeout, retry in 5 minutes...\n")
        time.sleep(60 * 5)
        continue
    except tweepy.error.TweepError as er:
        print('TweepError: ' + er.message)
        if er.message == 'Not authorized.':
            new_tweets = []
            break
        else:
            print(str(ntries))
            ntries +=1
            pass
    except:
        print "Unexpected error:", sys.exc_info()[0]
        new_tweets = []
        break
+4
source share

Source: https://habr.com/ru/post/1621135/


All Articles