Simple Tweets collection using Twitter’s Stream API and Java using Twitter4j

Fairly simple tweet collection (you will need to add to file/database/other storage) using Twitter’s stream API and Twitter4j

You need to add the Twitter4j core and stream Jars to your java application & you need to add in your own Twitter application tokens and user tokens (you should change this to not hardcode in the user tokens)

package com.crowley.simplestream;

import twitter4j.FilterQuery;
import twitter4j.Status;
import twitter4j.StatusDeletionNotice;
import twitter4j.StatusListener;
import twitter4j.TwitterStream;
import twitter4j.TwitterStreamFactory;
import twitter4j.User;
import twitter4j.conf.ConfigurationBuilder;

public class SimpleStream {
    public static void main(String[] args) {
        ConfigurationBuilder cb = new ConfigurationBuilder();
        cb.setDebugEnabled(true);
        cb.setOAuthConsumerKey("*************");
        cb.setOAuthConsumerSecret("*************");
        cb.setOAuthAccessToken("*************");
        cb.setOAuthAccessTokenSecret("*************");

        TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();

        StatusListener listener = new StatusListener() {

            @Override
            public void onException(Exception arg0) {
                // TODO Auto-generated method stub

            }

            @Override
            public void onDeletionNotice(StatusDeletionNotice arg0) {
                // TODO Auto-generated method stub

            }

            @Override
            public void onScrubGeo(long arg0, long arg1) {
                // TODO Auto-generated method stub

            }

            @Override
            public void onStatus(Status status) {
                User user = status.getUser();
                
                // gets Username
                String username = status.getUser().getScreenName();
                System.out.println(username);
                String profileLocation = user.getLocation();
                System.out.println(profileLocation);
                long tweetId = status.getId(); 
                System.out.println(tweetId);
                String content = status.getText();
                System.out.println(content +"\n");

            }

            @Override
            public void onTrackLimitationNotice(int arg0) {
                // TODO Auto-generated method stub

            }

        };
        FilterQuery fq = new FilterQuery();
    
        String keywords[] = {"ireland"};

        fq.track(keywords);

        twitterStream.addListener(listener);
        twitterStream.filter(fq);  

    }
}

162 thoughts on “Simple Tweets collection using Twitter’s Stream API and Java using Twitter4j”

  1. Great code!. Do you know a way to get ‘all’ tweets that have a certain hashtag? I mean, already tweeted, not ‘on the fly’ like this example.
    Thanks.

    1. As far as I know – no. You can get recent ones but not ones in the past. Not without going to one of the companies that allow you access to the Twitter stream fully like Gnip

  2. i have a problem
    Exception in thread “main” java.lang.IllegalArgumentException: Invalid access token format.
    at twitter4j.auth.AccessToken.(AccessToken.java:50)
    at twitter4j.auth.OAuthAuthorization.(OAuthAuthorization.java:69)
    at twitter4j.auth.AuthorizationFactory.getInstance(AuthorizationFactory.java:40)
    at twitter4j.TwitterStreamFactory.getInstance(TwitterStreamFactory.java:77)
    at coba2.SimpleStream.main(SimpleStream.java:30)
    Java Result: 1

    1. I’d guess that you don’t have the tokens set for your application – you can get tokens when you set up an application on Twitter’s development pages

    1. Two filters?
      You can track two words by String keywords[] = {“ireland”, “weather”}; or do you mean track “ireland” && “weather”?

  3. I’m trying to run this code having twitter4j-core-2.2.6.jar and twitter4j-stream-2.2.4.jar but this gives me error saying :
    1. Status Listener is not set StatusListener listener = new StatusListener()
    2. gives error at twitterStream.addListener(listener); – saying not a method of twitterStream.

    Please help as i’m jostling with this problem since last 2 days. Thank you.

    1. I’ll have a look tomorrow at my code and see if anything has changed within twitter4j to cause these errors

      DC

    2. Maybe the mismatch in versions is the problem? I am using 2.2.5 core and stream. Maybe use 2.2.6 core and stream?

      DC

  4. Hi,
    This example allows to filter tweets in real time, but how search for previous tweets
    (using streaming interface)?
    Thanks

    1. To search for older tweets you have to use the search api – i’ll see can i put together a similar sample for that in the next couple of days

      DC

  5. I am using twitter4j 3.0.3 and the classes StatusListener; TwitterStream;TwitterStreamFactory; are not getting recognized what wrong am i doing?

    1. Hi Gabriel,

      I changed my jars to Twitter4j 3.03 core and stream and it worked fine – I did have to add an unimplemented method for StatusListener but that was the only change I made,

      Regards,

      DC

  6. I am getting the error
    HTTP ERROR: 401
    Problem accessing ‘/1.1/statuses/filter.json’. Reason:
    Unauthorized

    TwitterStreamFactory twitterStreamFactory = new TwitterStreamFactory(configuration);
    TwitterStream twitterStreamClient = twitterStreamFactory.getInstance();
    FilterQuery filterQuery = new FilterQuery(followersId);
    twitterStreamClient.addListener(statusListener);
    twitterStreamClient.filter(filterQuery);

    But the same Configuration is working for twitter API

    1. a 401 means you can connect but you haven’t authenticated? Did you enter in your own secret keys?

      DC

  7. Hello David,

    Every time i am getting this in eclipse console
    [Fri May 31 11:01:02 IST 2013]Establishing connection.
    [Fri May 31 11:01:22 IST 2013]connect timed out
    [Fri May 31 11:01:22 IST 2013]Waiting for 16000 milliseconds
    [Fri May 31 11:01:38 IST 2013]Establishing connection.
    [Fri May 31 11:01:58 IST 2013]connect timed out
    [Fri May 31 11:01:58 IST 2013]Waiting for 16000 milliseconds
    [Fri May 31 11:02:14 IST 2013]Establishing connection.
    [Fri May 31 11:02:34 IST 2013]connect timed out
    [Fri May 31 11:02:34 IST 2013]Waiting for 16000 milliseconds
    [Fri May 31 11:02:50 IST 2013]Establishing connection.
    [Fri May 31 11:03:11 IST 2013]connect timed out
    [Fri May 31 11:03:11 IST 2013]Waiting for 16000 milliseconds
    [Fri May 31 11:03:27 IST 2013]Establishing connection.
    [Fri May 31 11:03:47 IST 2013]connect timed out

    Help me to fix this.

    1. Hi,

      Make sure you have internet?! 😛 and make sure your time on your computer is right as often it won’t allow a connection if it is wrong. Also make sure you are connecting to the right addresses

      DC

    2. Hi Karunjay Anand, did you found any fix for this.
      Even am facing with the same issue. I think its because of proxy.

        1. Hello Lalitha,

          I am using a school ubuntu server to collect tweets, and I have the same problem. Can you specify how you set the proxy? Thanks!

  8. Hey just used this as a template to start a project I’ve been wanting to do forever – found a site that charges at least 50 a month (at min) and you can only search for two hashtags – this option is infinitely more flexible and suitable for a custom application – just wanted to say thanks for the post, so awesome and I’ll check back if anyone needs help with theirs
    (was not paid to write this lol)

  9. Is there a way to retrieve tweets based on multiple keywords, like –
    String keywords[] = {“ireland”,”belgium”,”london”};
    fq.track(keywords);

    Pls let me know? I’m struggling to achieve this using Streaming API.

    1. Yes – a comma separated list should work like

      String keywords[] = {“ireland”, “kerry”};

      But the tweets don’t have to contain both Ireland & Kerry

        1. From looking at this -https://dev.twitter.com/docs/streaming-apis/parameters#track I am not sure if you can – you might have to use the search API. I’ll have to look into it more to be sure

  10. How does FilterQuery filter the streamed data? Coz every post returned does not contain my filter strings. And why does it sometimes return some post with ????? user and ????????? post?

    1. It isn’t a logical AND – it returns any tweet with string1 or string2 or string3.

      ???? could be an encoding problem?

  11. Thank you.
    I need to have something like a big corpus from old tweets and then streaming new tweets .
    I am confused .I looked it up but it seems that it needs some permissions.am I right?

    1. Not sure what you mean by permissions? permission to publish the tweets?

      Or do you mean authenticate your application/user?

  12. I am not sure too.But I think some kind of authentication from twitter website.
    anyways,the only thing I want to do is collect old tweets and streaming new ones which have a special hashtag and make a simple text file from all of them.can you save my life please?
    Thank you so much

    1. That is more difficult – you will need to keep track of “where” you are – so you don’t store duplicates. For that you might just use search and not use the streaming API at all.

      1. Could you be more specific ?I am new to these stuff.Could you send me some sort of code?
        Thank you so much

  13. Thank you for your code.I tried your code and the output is weird .here is a sample of output:

    19:12:05.229 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – User-Agent: twitter4j http://twitter4j.org/ /3.0.3
    19:12:05.229 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-Version: 3.0.3
    19:12:05.230 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Connection: close
    19:12:05.230 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Post Params: count=0&track=happy&stall_warnings=true
    19:12:05.300 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    19:12:05.300 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – Waiting for 16000 milliseconds
    19:12:05.300 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Waiting for 16000 milliseconds]
    19:12:21.301 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    19:12:21.301 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] INFO twitter4j.TwitterStreamImpl – Establishing connection.
    19:12:21.302 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Establishing connection]
    19:12:21.302 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Request:
    19:12:21.302 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – POST https://stream.twitter.com/1.1/statuses/filter.json
    19:12:21.303 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth base string: POST&https%3A%2F%2Fstream.twitter.com%2F1.1%2Fstatuses%2Ffilter.json&count%3D0%26oauth_consumer_key%3DVzdS7zf33IyW8IdFZYhHg%26oauth_nonce%3D1768804082%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1375927941%26oauth_token%3D1594456190-lXS9kTMkl0x7ttPxcr0RwIEjXix9k3g2xaGikiy%26oauth_version%3D1.0%26stall_warnings%3Dtrue%26track%3Dhappy
    19:12:21.304 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth signature: y6LmSWcLBff6pPxUQ5RZz636T0w=
    19:12:21.304 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Authorization: *******************************************************************************************************************************************************************************************************************************************************************************
    19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-3.0.3.xml
    19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client: Twitter4J
    19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Accept-Encoding: gzip
    19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – User-Agent: twitter4j http://twitter4j.org/ /3.0.3
    19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-Version: 3.0.3
    19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Connection: close
    19:12:21.306 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Post Params: count=0&track=happy&stall_warnings=true
    19:12:21.377 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    19:12:21.377 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – Waiting for 16000 milliseconds
    19:12:21.377 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Waiting for 16000 milliseconds]
    19:12:37.378 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    19:12:37.378 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] INFO twitter4j.TwitterStreamImpl – Establishing connection.
    19:12:37.378 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Establishing connection]
    19:12:37.379 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Request:
    19:12:37.379 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – POST https://stream.twitter.com/1.1/statuses/filter.json
    19:12:37.380 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth base string: POST&https%3A%2F%2Fstream.twitter.com%2F1.1%2Fstatuses%2Ffilter.json&count%3D0%26oauth_consumer_key%3DVzdS7zf33IyW8IdFZYhHg%26oauth_nonce%3D1789025394%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1375927957%26oauth_token%3D1594456190-lXS9kTMkl0x7ttPxcr0RwIEjXix9k3g2xaGikiy%26oauth_version%3D1.0%26stall_warnings%3Dtrue%26track%3Dhappy
    19:12:37.381 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth signature: g+U46S4Wet5ZN8Ru6BrY7Zo+rkY=
    19:12:37.381 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Authorization: ***********************************************************************************************************************************************************************************************************************************************************************************
    19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-3.0.3.xml
    19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client: Twitter4J
    19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Accept-Encoding: gzip
    19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – User-Agent: twitter4j http://twitter4j.org/ /3.0.3
    19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-Version: 3.0.3
    19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Connection: close
    19:12:37.383 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Post Params: count=0&track=happy&stall_warnings=true
    19:12:37.452 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    19:12:37.452 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – Waiting for 16000 milliseconds
    19:12:37.452 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Waiting for 16000 milliseconds]
    19:12:53.453 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    19:12:53.453 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] INFO twitter4j.TwitterStreamImpl – Establishing connection.
    19:12:53.453 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Establishing connection]
    19:12:53.454 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Request:
    19:12:53.454 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – POST https://stream.twitter.com/1.1/statuses/filter.json
    19:12:53.455 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth base string: POST&https%3A%2F%2Fstream.twitter.com%2F1.1%2Fstatuses%2Ffilter.json&count%3D0%26oauth_consumer_key%3DVzdS7zf33IyW8IdFZYhHg%26oauth_nonce%3D-171852571%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1375927973%26oauth_token%3D1594456190-lXS9kTMkl0x7ttPxcr0RwIEjXix9k3g2xaGikiy%26oauth_version%3D1.0%26stall_warnings%3Dtrue%26track%3Dhappy
    19:12:53.456 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth signature: qnPDnV9plkCmdzZzoTQa/7EDyWk=
    19:12:53.456 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Authorization: *********************************************************************************************************************************************************************************************************************************************************************************
    19:12:53.456 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-3.0.3.xml
    19:12:53.456 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client: Twitter4J
    19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Accept-Encoding: gzip
    19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – User-Agent: twitter4j http://twitter4j.org/ /3.0.3
    19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-Version: 3.0.3
    19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Connection: close
    19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Post Params: count=0&track=happy&stall_warnings=true
    19:12:53.528 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    19:12:53.529 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – Waiting for 16000 milliseconds
    19:12:53.529 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Waiting for 16000 milliseconds]

    1. Exit your IDE and run it again – seen this error message when you have already made a connection and it didn’t close correctly

  14. Hi DC,
    I tried the code and I am getting following result:

    [Wed Aug 21 10:58:49 IST 2013]Establishing connection.
    [Wed Aug 21 10:58:53 IST 2013]Extensions not allowed in v2 certificate
    [Wed Aug 21 10:58:53 IST 2013]Waiting for 250 milliseconds
    [Wed Aug 21 10:58:53 IST 2013]Establishing connection.
    [Wed Aug 21 10:58:54 IST 2013]Extensions not allowed in v2 certificate
    [Wed Aug 21 10:58:54 IST 2013]Waiting for 500 milliseconds
    [Wed Aug 21 10:58:55 IST 2013]Establishing connection.
    [Wed Aug 21 10:58:55 IST 2013]Extensions not allowed in v2 certificate
    [Wed Aug 21 10:58:55 IST 2013]Waiting for 1000 milliseconds
    [Wed Aug 21 10:58:56 IST 2013]Establishing connection.
    [Wed Aug 21 10:58:57 IST 2013]Extensions not allowed in v2 certificate
    [Wed Aug 21 10:58:57 IST 2013]Waiting for 2000 milliseconds
    [Wed Aug 21 10:58:59 IST 2013]Establishing connection.
    [Wed Aug 21 10:59:00 IST 2013]Extensions not allowed in v2 certificate
    [Wed Aug 21 10:59:00 IST 2013]Waiting for 4000 milliseconds
    [Wed Aug 21 10:59:04 IST 2013]Establishing connection.
    [Wed Aug 21 10:59:05 IST 2013]Extensions not allowed in v2 certificate
    [Wed Aug 21 10:59:05 IST 2013]Waiting for 8000 milliseconds

    Please Help!

  15. How can we get the latest tweet from public twitter profile using the streaming api? How do i filter content to get the tweets from a specific public twitter profile.

      1. Yup, it works. Problem was with the #tag i gave to search. At that moment, there were no recent tweets related to that #tag. Hence, thought it doesn’t work with hastags.
        Thanks 🙂

  16. Hi David
    That’s a great piece of code . I am completely new to this . I tried the code . I wanted to stop the code after a particular interval of time. But unable to do so. Any help is appreciated.
    Thanks

    1. You could loop until you have a certain amount of tweets? or you could use time to do it for so many seconds. Depends on what you are trying to do

  17. Hello David,

    there is a compilation error with your code:

    SimpleStream.java:22: error: is not abstract and does not override abstract method onStallWarning(StallWarning) in StatusListener
    StatusListener listener = new StatusListener() {
    ^
    Can it be fixed?

  18. Hi. I’m very comfortable with Java and have done some successful testing with Twitter4J from my IDE, but I’m at a loss when searching for resources on how to deploy a Java that could monitor a twitter feed. Can you point me to some useful resources (i.e. where/how to host, deployment how-tos)? Thanks

  19. Hello:

    Have you found any way to run multiple listeners with just one TwitterStream connection. ?

    Trying to creatively get around some of the limitations that Twitter is imposing, such as only one concurrent TwitterStream connection, only 400 keywords via the fq.track(keywords) method, etc.

    I tried creating 2 separate listeners in one method but since they are both connected to the same TwitterStream it treated it as if it were two separate connections and disconnects.

    Im beginning to work on a multi-threaded approach to see it that has any merit.
    Just curious if you have any ideas.

    Thanks

  20. Hey, thanks for your code. When I ran your code am getting the error as
    ” sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”
    Am using twitter4j-3.0.5, NetBeansIDE7.4, https internet connection in Windows7. Could you pls help me to resolve this issue?

  21. Hey David i just copied your code… But i still get this error below… It looks like, i miss some methods in my Jar-Files. Can u check this out and tell me what i do wrong? If it’s juste the Jar-Files… Can send my the right one.
    I use :
    twitter4j-stream-3.0.3.jar
    twitter4j-core-4.0.1.jar

    Exception in thread “main” java.lang.NoClassDefFoundError: twitter4j/internal/http/HttpClientWrapperConfiguration
    at twitter4j.TwitterStreamFactory.(TwitterStreamFactory.java:40)
    at twitter.SimpleStream.main(SimpleStream.java:23)
    Caused by: java.lang.ClassNotFoundException: twitter4j.internal.http.HttpClientWrapperConfiguration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    … 2 more

    1. At a guess I would say that using twitter4j-stream-3.0.3.jar
      twitter4j-core-4.0.1.jar non matching jars would be the problem if you download Twitter4j then the jars should be of the same version?

      1. I guessed it too, but i also tried each at the time… Then i got other issues: it doesn’t recognize some imported sources. Can u send me the Referenced Libraries u used to compile your code? It’ll be nice, if i can get your code running.

        thanks.

  22. I have got this problem fixed by using these 2 Jar-Files:
    – twitter4j-stream-3.0.3.jar
    – twitter4j-core-3.0.5.jar

    Now i get some Twitter-API struggles. And i don’t know what wrong with that problem… It looks like my OAuthConsumerKey, OAuthConsumerSecret, … are false. But that not true, then i used them with the Rest-API and they worked very well.

    Please help!

    [Fri Apr 04 12:51:54 CEST 2014]Establishing connection.
    [Fri Apr 04 12:51:57 CEST 2014]401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.
    \n\n\nError 401 Unauthorized

    HTTP ERROR: 401
    Problem accessing ‘/1.1/statuses/filter.json’. Reason:
    Unauthorized

    [Fri Apr 04 12:51:57 CEST 2014]Waiting for 10000 milliseconds
    [Fri Apr 04 12:52:07 CEST 2014]Establishing connection.
    [Fri Apr 04 12:52:07 CEST 2014]401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.
    \n\n\nError 401 Unauthorized

    HTTP ERROR: 401
    Problem accessing ‘/1.1/statuses/filter.json’. Reason:
    Unauthorized

    [Fri Apr 04 12:52:07 CEST 2014]Waiting for 20000 milliseconds

  23. I am getting the below two lines

    [Wed Apr 23 14:54:48 IST 2014]Will use class twitter4j.internal.logging.StdOutLoggerFactory as logging factory.
    [Wed Apr 23 14:54:48 IST 2014]Will use twitter4j.internal.http.HttpClientImpl as HttpClient implementation.

  24. Thanks for the valuable information!
    I want to know we can limit the number of tweets we get by setting the listener. I mean for a given filter, the onStatus method keeps getting called infinitely. Can we like limit the onStatus method being called for a fixed number of times without using System.exit? Thanks!

    1. You could build some kind of Tweet counter and stop the listener after that amount. Have never tried it. You could also use a Timer and only collect for a set time scale.

  25. Respected Sir,

    First of all thanks for your code. But I have a problem. When I run the program it shows exceptions like

    Exception in thread “main” java.lang.NoClassDefFoundError: SimpleStream (wrong name: com/crowley/simplestream/SimpleStream)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:472)

    I used twitter4j. Please help me.

    1. Looking at this line Exception in thread “main” java.lang.NoClassDefFoundError: SimpleStream (wrong name: com/crowley/simplestream/SimpleStream) i wonder is this the problem?

  26. HI
    thanks for the code, could you just explain in detail, how to link the twitter4j files to the program

  27. i want change the keyword after every 600 seconds but i m not able to do it.Whenever i cleanup and shutdown streamfactory after every keyword it says inflator has been closed.
    It only able to fetch 1st keyword.

  28. I am getting Estabilishing Connection. Waiting for 16000 milliseconds continously.
    How to bypass proxy in Ubantu ?
    Please experts share your ideas…..

  29. Hi

    This was an awesome tutorial for newbies like me.
    I want to store the output Tweet stream to a JSON file,could you please share any code for that?

    Thanks in Advance!

  30. Hi
    thanks for this great post. I have a little different problem here. I am trying to get all tweets that are streaming and print it in real time using twitter4j. This means that i should use the listener without any filter, but i don t know how to implement this. Can you help me?

  31. Hi, thanks for the codes… But there’re some problems with my output like:

    [Thu Oct 16 12:30:39 SGT 2014]Establishing connection.
    [Thu Oct 16 12:30:41 SGT 2014]401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.
    \n\n\nError 401 Unauthorized

    HTTP ERROR: 401
    Problem accessing ‘/1.1/statuses/filter.json’. Reason:
    Unauthorized

    I’ve applied for the keys and tokens yesterday, but I still get a 401. Can you please help me figure it out? Thank you.

  32. can anyone tell me how to add jar files and setting path through command prompt..please give me the clear steps about what to do after downloading twitter4j 4.0.2 API.Thanks!

  33. I use your code, but have this error:

    Exception in thread “main” java.lang.NoClassDefFoundError: twitter4j/internal/http/HttpClientWrapperConfiguration
    at twitter4j.TwitterStreamFactory.(TwitterStreamFactory.java:40)
    at javaapplication3.JavaApplication3.main(JavaApplication3.java:35)
    Caused by: java.lang.ClassNotFoundException: twitter4j.internal.http.HttpClientWrapperConfiguration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    … 2 more
    Java Result: 1

    The line which caused the error is:
    TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();

    Thank you very much!

  34. Hey, great code helped me a lot! I wanted to know that is there any way to insert these incoming tweet stream to directly into a mysql table using jdbc?

  35. Hello DC,

    Any thoughts on how to stream tweets with just the geo-location (geo-boxing) and not using the search keywords inside the track method.

    Basically want to stream tweets from a particular country

  36. How can i save the output as a .txt file and use it rather than seeing in console..I am new to this help me.

  37. Thank you.But When i run your above program and save it in a file it overwrites the previous tweets.Any Solution for that?? I need to save all the streaming Tweets in a .txt file

    1. Hey, Aravind you need to change the name of txt file every time in program if you don’t want your collected tweets to be overwritten. Whenever you stop the streaming and a bunch of tweets are collected and next time when you start again to collect new set of tweets just change the name of text file to whatever you want to. This happens because in program PrintStream will first find whether the file with given name on given path exists or not. If file is present then it will overwrite the contents, if not then it will make a new txt file and writes new content(here tweets) in newly created text file. Hope this helps.

  38. hi,
    i am using eclipse tool and twitter 4j api to collect the tweets.How to store the collected tweets in excel sheet ad how to apply the sentiment analysis.please do let me know how to do this.

    thanks in advance

    1. You wouldn’t save them in an Excel format. Save them in CSV format. Then you can pick an already existing sentiment analysis package and use that. There is a lot of examples and lots of libraries depending on what language you are comfortable with (Python or R maybe)

      1. Sir i did’nt get how to store the data in CSV format please help me. I have used lexicon based approach to calucate polarity of tweets.

        Thank you

  39. crowley sir pls help its very urgent i have hadoop 1.2.1 and ubuntu 14.04 flume 1.6 i m trying to get twitter data in hdfs but

    previously i got errror while running flume agent “NO SUCH METHOD FOUND ” which i corrected by googling by replacing twitter jar of version 3..*.* with 2.*.*
    BUT NOW I M GETTING ERROR “JAVA.LANG.NULLPOINTEREXCEPTION consumer secret is null”
    pls i have tried a lot u caN Screenshots are below
    ALSO TO TELL U THAT PREVIOUSLY I GOT ERROR COULD NOT LOAD MAIN CLASS SO I UNSET FLUME HOME

    21 Mar 2016 07:51:25,851 ERROR [lifecycleSupervisor-1-5] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) – Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } – Exception follows.
    java.lang.NullPointerException: consumer secret is null
    at twitter4j.TwitterBaseImpl.setOAuthConsumer(TwitterBaseImpl.java:250)
    at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:129)
    at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
    at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

  40. pls help me i have found it everywhere i can find no answer pls i will be shattered i don’t get help this is my twitter.conf
    TwitterAgent.sources= Twitter
    TwitterAgent.channels= MemChannel
    TwitterAgent.sinks=HDFS
    TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
    TwitterAgent.sources.Twitter.channels= MemChannel

    TwitterAgent.sources.Twitter.consumerSecret=
    XXX
    TwitterAgent.sources.Twitter.consumerKey= XXX
    TwitterAgent.sources.Twitter.accessTokenSecret= XXXX
    TwitterAgent.sources.Twitter.accessToken=
    XXXX

    TwitterAgent.sources.Twitter.keywords= samsung galaxy,samsung phone launch,samsung mobile

    TwitterAgent.sinks.HDFS.channel=MemChannel
    TwitterAgent.sinks.HDFS.type=hdfs
    TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:9000/winit/%Y/%m/%d/%H/
    TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
    TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
    TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
    TwitterAgent.sinks.HDFS.hdfs.rollSize=0
    TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
    TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
    TwitterAgent.channels.MemChannel.type=memory
    TwitterAgent.channels.MemChannel.capacity=10000
    TwitterAgent.channels.MemChannel.transactionCapacity=1000

  41. Can you send me code to print all tweets on son format;
    What must change on this to code to work;
    Thanks you very much

          1. Thank you very much!
            But i have problem with the raw “for (Tweet tweet : result.getTweets())”
            and the problem is the “Tweet” i can’t compile it. Why?

          2. Any error message? And if you are using the code from StackOverFlow then you could just ask there?

  42. cannot find symbol
    symbol: class Tweet
    location: class TwitterApp2
    Can use functional operators
    And when i press to fix the bug :
    Create a local variable “Tweet”
    Use functional Operators
    But nothing works!
    if you can help me to fix this bug and start download data from twitter in json format.
    Thank you

  43. Thank you very much for this example, David!
    I had to set proxy host and port to make this program work on my machine:
    cb.setHttpProxyHost(“proxy”);
    cb.setHttpProxyPort(8080);

    1. I had also set VM args and I think that did the trick instead of cb.setHttp… methods:
      -Dhttp.proxyPort=8080 -Dhttp.proxyHost=proxy -Dhttps.proxyPort=8080 -Dhttps.proxyHost=proxy

  44. Hi Sadhana,

    Try out these lines just above ConfigurationBuilder in the code. I have not tried this in csv format, but still try this. When I was working with this, I stored data in txt file. While storing you might face mapping issues.

    FileOutputStream f = new FileOutputStream(“C:\\your_path\\your_file_name.csv”);
    System.setOut(new PrintStream(f));

  45. Hi Sir i have a problem ,,

    Exception in thread “main” java.lang.RuntimeException: Uncompilable source code – is not abstract and does not override abstract method onStallWarning(twitter4j.StallWarning) in twitter4j.StatusListener
    at Simple.main(Simple.java:31)

    how o fix it ?

  46. Hello sir,

    I have a question. I was able to configure everything successfully and collect tweets based on hashtags that I provided in my default list of keywords. Now besides that I am also collecting other hashtags that appear in tweets, the ones that are not in my default keyword list, and then I add them to my default list so that I can receive tweets with those hashtags too.
    But my problem is I am focusing on specific event (let’s say olympics) and sometimes I get some hashtags that are not really related to olympics, but they were tweeted with some of the hashtags from my default list. Do you have any idea maybe from statistical or algorithmic point, how could I solve this problem, or I mean how can I decide whether specific hashtags should be added to my default list of keywords or not (is it relevant to the event for which I am trying to collect data)?

    I know this might not be related to this discussion, but I would appreciate any kind of advice or suggestion.

    Thanks in advance.

    1. This is much more of a research question then a coding question. I think looking at your data would help – collect specific tweets with the hashtag. Look at other hashtag use then evolve the system. Spammers seem to use hashtags a lot – but they often use just trending topics so you could examine both using inclusion of hashtags and then removing tweets that have unrelated trending hashtags in them. Again this could be difficult as hashtags could evolve. Semi related is work by Pavan Kapanipathi & Fabrizio Orlandi http://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=2025&context=knoesis using semantic technologies to take tweet content and tie them to external concepts. This might help

  47. Thanks for the blog…sir…
    I am getting rate limit exception ….
    Could you please tell me how can I avoid this….

    1. Read the Twitter API guidelines – depends on what you want to do. You could do a request (timed) to stop the rate limiting

  48. I am getting the following error :
    I have added java-json.jar

    [Mon Feb 27 12:14:10 IST 2017]Establishing connection.
    Exception in thread “Twitter Stream consumer-1[Establishing connection]” java.lang.NoClassDefFoundError: twitter4j/internal/org/json/JSONException
    at twitter4j.TwitterStreamImpl.getFilterStream(TwitterStreamImpl.java:290)
    at twitter4j.TwitterStreamImpl$7.getStream(TwitterStreamImpl.java:279)
    at twitter4j.TwitterStreamImpl$7.getStream(TwitterStreamImpl.java:277)
    at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:427)
    Caused by: java.lang.ClassNotFoundException: twitter4j.internal.org.json.JSONException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    … 4 more

    1. You could possibly be using mismatched versions? Check if your streaming and search versions of twitter4j are the same

      1. I am using different versions of stream and core files .
        If I use the latest version 4.0.4 I get errors at .addListener.
        All the errors get resolved after adding “twitter4j-core-3.0.4” , “twitter4j-stream-2.2.5” , “twitter-4j”.

        But I get the following error now :

        [Mon Feb 27 15:57:10 IST 2017]Establishing connection.
        [Mon Feb 27 15:57:13 IST 2017]Connection established.
        [Mon Feb 27 15:57:13 IST 2017]Receiving status stream.

        Exception in thread “Twitter4J Async Dispatcher[0]” java.lang.NoSuchMethodError: twitter4j.json.JSONObjectType.determine(Ltwitter4j/internal/org/json/JSONObject;)Ltwitter4j/json/JSONObjectType;
        at twitter4j.AbstractStreamImplementation$1.run(AbstractStreamImplementation.java:100)
        at twitter4j.internal.async.ExecuteThread.run(DispatcherImpl.java:116)

    1. 4.0.4 doesn’t work well.
      I tried using 3.0.4 core and stream jar files and it worked well for me .
      Getting all the live tweets.
      Thanks alot .
      It would be really helpful if you would describe how the code works exactly.

      1. I get the following response for content :
        RT @farukonalan: 2015’de Daesh’in infaz görüntülerini paylasan Le Pen’in dokunulmazligi “terör propagandasi” kapsaminda kaldiriliyor. https…

        Here the URL received is incomplete.This happens alot of times.I wanted to know what is the issue with the data?
        Why do I get incomplete tweets ?

        1. Ask Twitter! or the Twitter4j developers – or start looking at the raw JSON data returned by the request.

  49. Hello,

    I have implemented this and it works like charm. Now if I want to dynamically add more user to follow, how should I go about it? ie The updated list of users to follow gets stored in DB and from there if a new user is added or an old user got removed I need to update my follow predicates and restart the stream in graceful manner. Can you provide a code sample for this

    Regards
    Rahul

    1. I could – but I am not a contract developer. Probably adding them daily to a list/db and restarting the process when you have updated the list of users you want to follow.

Leave a Reply