Twitter4j since id is not updating

Question

I am trying to use from_id to get tweets using twitter search api. Below is my code, here I create a map of the request object and with id. I default from id to 0, and my goal is to update the id with id every time I run a request. So the next time I run the request, it will not receive the same tweets and should start with the last tweet.

import java.io.{PrintWriter, StringWriter}
import java.util.Properties
import com.google.common.io.Resources
import twitter4j._
import scala.collection.JavaConversions._
// reference: http://bcomposes.com/2013/02/09/using-twitter4j-with-scala-to-access-streaming-tweets/
object Util {
    val props = Resources.getResource("twitter4j.props").openStream()
    val properties = new Properties()
    properties.load(props)

    val config = new twitter4j.conf.ConfigurationBuilder()
        .setDebugEnabled(properties.getProperty("debug").toBoolean)
        .setOAuthConsumerKey(properties.getProperty("consumerKey"))
        .setOAuthConsumerSecret(properties.getProperty("consumerSecret"))
        .setOAuthAccessToken(properties.getProperty("accessToken"))
        .setOAuthAccessTokenSecret(properties.getProperty("accessTokenSecret"))
    val tempKeys =List("Yahoo","Bloomberg","Messi", "JPM Chase","Facebook")
    val sinceIDmap : scala.collection.mutable.Map[String, Long] = collection.mutable.Map(tempKeys map { ix => s"$ix" -> 0.toLong } : _*)
    //val tweetsMap: scala.collection.mutable.Map[String, String]
    val configBuild = (config.build())
    val MAX_TWEET=100
    getTweets()

    def getTweets(): Unit ={
        sinceIDmap.keys.foreach((TickerId) => getTweets(TickerId))
    }

    def getTweets(TickerId: String): scala.collection.mutable.Map[String, scala.collection.mutable.Buffer[String]] = {
        println("Search key is:"+TickerId)
        var tweets = scala.collection.mutable.Map[String, scala.collection.mutable.Buffer[String]]()
        try {
            val twitter: Twitter = new TwitterFactory(configBuild).getInstance
            val query = new Query(TickerId)
            query.setSinceId(sinceIDmap.get(TickerId).get)
            query.setLang("en")
            query.setCount(MAX_TWEET)
            val result = twitter.search(query)
            tweets += ( TickerId -> result.getTweets().map(_.getText))

            //sinceIDmap(TickerId)=result.getSinceId
            println("-----------Since id is :"+result.getSinceId )
            //println(tweets)
        }
        catch {
            case te: TwitterException =>
                println("Failed to search tweets: " + te.getMessage)
        }
        tweets
    }
}

object StatusStreamer {
    def main(args: Array[String]) {
        Util
    }
}

output:

Search key is:Yahoo    
log4j:WARN No appenders could be found for logger (twitter4j.HttpClientImpl).
log4j:WARN Please initialize the log4j system properly.
-----------Since id is :0
Search key is:JPM Chase
-----------Since id is :0
Search key is:Facebook
-----------Since id is :0
Search key is:Bloomberg
-----------Since id is :0
Search key is:Messi
-----------Since id is :0

The problem is that when I try to print id id after running the request, it gives the same value as at the beginning. Can someone tell me what I'm doing wrong here? or if my approach is wrong, someone can use some other approach if they know that it will work here.

thank

Answer 1

Twitter API since_id, . , QueryResult.getSinceId , Query.

sinceId .

sinceIDmap(TickerId) = result.getTweets().max(Ordering.by(_.getId)).getId

, , since_id max_id. , .

Answer 2

, , since_id . . , Timeline. , . , since_id ( , GET search / tweets ). max_id , . , , , since_id / ( ). API docs , since_id , max_id, . since_id, , , since_id . - , . , since_id , . since_id, / , , , , , since_id. , . , , :

since_id , . , Tweet 10 ( - ). , since_id .

, , , . , max_id .

max_id . , , max_id ( max_id, max_id , ). max_id , / . , .

since_id , . , - t0 . , - id0. , . , . , t1, , . t0 t1 . , t1, id0 ( t0). id0 since_id .. , since_id , , , , , id0 ( , ), since_id . , since_id , max_id .

. Timeline. , " max_id" , " from_id " - . , since_id .

Twitter4J Java, , , :

// Make sure this is initialized correctly.
Twitter twitter;

/**
 * Searches and prints tweets starting from now and going back to the past.
 * 
 * @param q
 *            the search query, e.g. "#yolo"
 */
private void searchAndPrintTweets(String q) throws TwitterException {
    // `max_id` needed by `GET search/tweets`. If it is 0 (first iteration),
    // it will not be used for the query.
    long maxId = 0;
    // Let us assume that it will run forever.
    while (true) {
        Query query = new Query();
        query.setCount(100);
        query.setLang("en");
        // Set `max_id` as an inclusive upper limit, unless this is the
        // first iteration. If this is the first iteration (maxId == 0), the
        // freshest/latest tweets will come.
        if (maxId != 0)
            query.setMaxId(maxId);
        QueryResult qr = twitter.search(query);
        printTweets(qr.getTweets());
        // For next iteration. Decrement smallest ID by 1, so that we will
        // not get the oldest tweet of this iteration in the next iteration
        // as well, since `max_id` is inclusive.
        maxId = calculateSmallestId(qr.getTweets()) - 1;
    }
}

/**
 * Calculates the smallest ID among a list of tweets.
 * 
 * @param tweets
 *            the list of tweets
 * @return the smallest ID
 */
private long calculateSmallestId(List<Status> tweets) {
    long smallestId = Long.MAX_VALUE;
    for (Status tweet : tweets) {
        if (tweet.getId() < smallestId)
            smallestId = tweet.getId();
    }
    return smallestId;
}

/**
 * Prints the content of the tweets.
 * 
 * @param tweets
 *            the tweets
 */
private void printTweets(List<Status> tweets) {
    for (Status tweet : tweets) {
        System.out.println(tweet.getText());
    }
}

, (, ), since_id, .

Answer 3

, , sinceIDmap. :

//sinceIDmap(TickerId)=result.getSinceId

, since_id 0.

, , Twitter4J SearchTweets GitHub.

Answer 4

Since_id max_id - , , API. :

since_id - , , ( , ) . , API. Tweets from_id, from_id . max_id - ( ) ID. , , , .

count - , , 200.

Unfortunately, the API will not return exactly what you want - you cannot specify the date / time when requesting user_timeline - although you can specify it when using the search API. In any case, if you need to use user_timeline, you will need to poll the API, collect tweets, find out if they match your parameters, and then calculate the statistics accordingly.