Fairly simple tweet collection (you will need to add to file/database/other storage) using Twitter’s stream API and Twitter4j
You need to add the Twitter4j core and stream Jars to your java application & you need to add in your own Twitter application tokens and user tokens (you should change this to not hardcode in the user tokens)
package com.crowley.simplestream; import twitter4j.FilterQuery; import twitter4j.Status; import twitter4j.StatusDeletionNotice; import twitter4j.StatusListener; import twitter4j.TwitterStream; import twitter4j.TwitterStreamFactory; import twitter4j.User; import twitter4j.conf.ConfigurationBuilder; public class SimpleStream { public static void main(String[] args) { ConfigurationBuilder cb = new ConfigurationBuilder(); cb.setDebugEnabled(true); cb.setOAuthConsumerKey("*************"); cb.setOAuthConsumerSecret("*************"); cb.setOAuthAccessToken("*************"); cb.setOAuthAccessTokenSecret("*************"); TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance(); StatusListener listener = new StatusListener() { @Override public void onException(Exception arg0) { // TODO Auto-generated method stub } @Override public void onDeletionNotice(StatusDeletionNotice arg0) { // TODO Auto-generated method stub } @Override public void onScrubGeo(long arg0, long arg1) { // TODO Auto-generated method stub } @Override public void onStatus(Status status) { User user = status.getUser(); // gets Username String username = status.getUser().getScreenName(); System.out.println(username); String profileLocation = user.getLocation(); System.out.println(profileLocation); long tweetId = status.getId(); System.out.println(tweetId); String content = status.getText(); System.out.println(content +"\n"); } @Override public void onTrackLimitationNotice(int arg0) { // TODO Auto-generated method stub } }; FilterQuery fq = new FilterQuery(); String keywords[] = {"ireland"}; fq.track(keywords); twitterStream.addListener(listener); twitterStream.filter(fq); } }
Very nice exemple thank’s so much
twitterStream.addListener(listener) method not found error
Same problem here. How o fix it?
Make sure you have the Twitter4j jars added to the project
I am getting the same error even after adding jars. Any clue?
Great code!. Do you know a way to get ‘all’ tweets that have a certain hashtag? I mean, already tweeted, not ‘on the fly’ like this example.
Thanks.
As far as I know – no. You can get recent ones but not ones in the past. Not without going to one of the companies that allow you access to the Twitter stream fully like Gnip
i have a problem
Exception in thread “main” java.lang.IllegalArgumentException: Invalid access token format.
at twitter4j.auth.AccessToken.(AccessToken.java:50)
at twitter4j.auth.OAuthAuthorization.(OAuthAuthorization.java:69)
at twitter4j.auth.AuthorizationFactory.getInstance(AuthorizationFactory.java:40)
at twitter4j.TwitterStreamFactory.getInstance(TwitterStreamFactory.java:77)
at coba2.SimpleStream.main(SimpleStream.java:30)
Java Result: 1
I’d guess that you don’t have the tokens set for your application – you can get tokens when you set up an application on Twitter’s development pages
TY so much But….
How to use 2 filters?
Two filters?
You can track two words by String keywords[] = {“ireland”, “weather”}; or do you mean track “ireland” && “weather”?
I’m trying to run this code having twitter4j-core-2.2.6.jar and twitter4j-stream-2.2.4.jar but this gives me error saying :
1. Status Listener is not set StatusListener listener = new StatusListener()
2. gives error at twitterStream.addListener(listener); – saying not a method of twitterStream.
Please help as i’m jostling with this problem since last 2 days. Thank you.
I’ll have a look tomorrow at my code and see if anything has changed within twitter4j to cause these errors
DC
Maybe the mismatch in versions is the problem? I am using 2.2.5 core and stream. Maybe use 2.2.6 core and stream?
DC
Hi,
This example allows to filter tweets in real time, but how search for previous tweets
(using streaming interface)?
Thanks
To search for older tweets you have to use the search api – i’ll see can i put together a similar sample for that in the next couple of days
DC
hey did you find any soluiton? Why you cant publish on this page?
Because there is lots of other examples of it if you search online – http://stackoverflow.com/questions/13545936/twitter4j-search-for-public-tweets
Nice man. Very good, and it works perfectly. Thank you very much.
I am using twitter4j 3.0.3 and the classes StatusListener; TwitterStream;TwitterStreamFactory; are not getting recognized what wrong am i doing?
Hi Gabriel,
I changed my jars to Twitter4j 3.03 core and stream and it worked fine – I did have to add an unimplemented method for StatusListener but that was the only change I made,
Regards,
DC
I am getting the error
HTTP ERROR: 401
Problem accessing ‘/1.1/statuses/filter.json’. Reason:
Unauthorized
TwitterStreamFactory twitterStreamFactory = new TwitterStreamFactory(configuration);
TwitterStream twitterStreamClient = twitterStreamFactory.getInstance();
FilterQuery filterQuery = new FilterQuery(followersId);
twitterStreamClient.addListener(statusListener);
twitterStreamClient.filter(filterQuery);
But the same Configuration is working for twitter API
a 401 means you can connect but you haven’t authenticated? Did you enter in your own secret keys?
DC
Hello David,
Every time i am getting this in eclipse console
[Fri May 31 11:01:02 IST 2013]Establishing connection.
[Fri May 31 11:01:22 IST 2013]connect timed out
[Fri May 31 11:01:22 IST 2013]Waiting for 16000 milliseconds
[Fri May 31 11:01:38 IST 2013]Establishing connection.
[Fri May 31 11:01:58 IST 2013]connect timed out
[Fri May 31 11:01:58 IST 2013]Waiting for 16000 milliseconds
[Fri May 31 11:02:14 IST 2013]Establishing connection.
[Fri May 31 11:02:34 IST 2013]connect timed out
[Fri May 31 11:02:34 IST 2013]Waiting for 16000 milliseconds
[Fri May 31 11:02:50 IST 2013]Establishing connection.
[Fri May 31 11:03:11 IST 2013]connect timed out
[Fri May 31 11:03:11 IST 2013]Waiting for 16000 milliseconds
[Fri May 31 11:03:27 IST 2013]Establishing connection.
[Fri May 31 11:03:47 IST 2013]connect timed out
Help me to fix this.
Hi,
Make sure you have internet?! 😛 and make sure your time on your computer is right as often it won’t allow a connection if it is wrong. Also make sure you are connecting to the right addresses
DC
Hi Karunjay Anand, did you found any fix for this.
Even am facing with the same issue. I think its because of proxy.
Hurray!!! i fixed it by setting the proxy 🙂
Hello Lalitha,
I am using a school ubuntu server to collect tweets, and I have the same problem. Can you specify how you set the proxy? Thanks!
You would be in corporate network. Switch to Simple internet.
HI HOW TO ADD BOUNDING BOX WITH TRACK.???
Do you mean a geolocated bounding box?
Hey just used this as a template to start a project I’ve been wanting to do forever – found a site that charges at least 50 a month (at min) and you can only search for two hashtags – this option is infinitely more flexible and suitable for a custom application – just wanted to say thanks for the post, so awesome and I’ll check back if anyone needs help with theirs
(was not paid to write this lol)
many thanks!!! 🙂
It was very helpful. Thanks!
Is there a way to retrieve tweets based on multiple keywords, like –
String keywords[] = {“ireland”,”belgium”,”london”};
fq.track(keywords);
Pls let me know? I’m struggling to achieve this using Streaming API.
Yes – a comma separated list should work like
String keywords[] = {“ireland”, “kerry”};
But the tweets don’t have to contain both Ireland & Kerry
Any idea how to search for tweets that contain both Ireland & Kerry?
Any idea how to make it search for both Ireland and Kerry?
From looking at this -https://dev.twitter.com/docs/streaming-apis/parameters#track I am not sure if you can – you might have to use the search API. I’ll have to look into it more to be sure
Thank you so much!!!
How does FilterQuery filter the streamed data? Coz every post returned does not contain my filter strings. And why does it sometimes return some post with ????? user and ????????? post?
It isn’t a logical AND – it returns any tweet with string1 or string2 or string3.
???? could be an encoding problem?
Thanks DC for this code! I would like to simply get a specific user’s tweets from the streaming API. How would I change the code to do so? Many tks!
There used to be a method called “getUserTimeLine” but might be deprecated?
Thank you.
I need to have something like a big corpus from old tweets and then streaming new tweets .
I am confused .I looked it up but it seems that it needs some permissions.am I right?
Not sure what you mean by permissions? permission to publish the tweets?
Or do you mean authenticate your application/user?
I am not sure too.But I think some kind of authentication from twitter website.
anyways,the only thing I want to do is collect old tweets and streaming new ones which have a special hashtag and make a simple text file from all of them.can you save my life please?
Thank you so much
That is more difficult – you will need to keep track of “where” you are – so you don’t store duplicates. For that you might just use search and not use the streaming API at all.
Could you be more specific ?I am new to these stuff.Could you send me some sort of code?
Thank you so much
it works, awesome. thanks for sharing the code 🙂
Thank you for your code.I tried your code and the output is weird .here is a sample of output:
19:12:05.229 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – User-Agent: twitter4j http://twitter4j.org/ /3.0.3
19:12:05.229 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-Version: 3.0.3
19:12:05.230 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Connection: close
19:12:05.230 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Post Params: count=0&track=happy&stall_warnings=true
19:12:05.300 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
19:12:05.300 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – Waiting for 16000 milliseconds
19:12:05.300 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Waiting for 16000 milliseconds]
19:12:21.301 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
19:12:21.301 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] INFO twitter4j.TwitterStreamImpl – Establishing connection.
19:12:21.302 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Establishing connection]
19:12:21.302 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Request:
19:12:21.302 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – POST https://stream.twitter.com/1.1/statuses/filter.json
19:12:21.303 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth base string: POST&https%3A%2F%2Fstream.twitter.com%2F1.1%2Fstatuses%2Ffilter.json&count%3D0%26oauth_consumer_key%3DVzdS7zf33IyW8IdFZYhHg%26oauth_nonce%3D1768804082%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1375927941%26oauth_token%3D1594456190-lXS9kTMkl0x7ttPxcr0RwIEjXix9k3g2xaGikiy%26oauth_version%3D1.0%26stall_warnings%3Dtrue%26track%3Dhappy
19:12:21.304 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth signature: y6LmSWcLBff6pPxUQ5RZz636T0w=
19:12:21.304 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Authorization: *******************************************************************************************************************************************************************************************************************************************************************************
19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-3.0.3.xml
19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client: Twitter4J
19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Accept-Encoding: gzip
19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – User-Agent: twitter4j http://twitter4j.org/ /3.0.3
19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-Version: 3.0.3
19:12:21.305 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Connection: close
19:12:21.306 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Post Params: count=0&track=happy&stall_warnings=true
19:12:21.377 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
19:12:21.377 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – Waiting for 16000 milliseconds
19:12:21.377 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Waiting for 16000 milliseconds]
19:12:37.378 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
19:12:37.378 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] INFO twitter4j.TwitterStreamImpl – Establishing connection.
19:12:37.378 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Establishing connection]
19:12:37.379 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Request:
19:12:37.379 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – POST https://stream.twitter.com/1.1/statuses/filter.json
19:12:37.380 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth base string: POST&https%3A%2F%2Fstream.twitter.com%2F1.1%2Fstatuses%2Ffilter.json&count%3D0%26oauth_consumer_key%3DVzdS7zf33IyW8IdFZYhHg%26oauth_nonce%3D1789025394%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1375927957%26oauth_token%3D1594456190-lXS9kTMkl0x7ttPxcr0RwIEjXix9k3g2xaGikiy%26oauth_version%3D1.0%26stall_warnings%3Dtrue%26track%3Dhappy
19:12:37.381 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth signature: g+U46S4Wet5ZN8Ru6BrY7Zo+rkY=
19:12:37.381 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Authorization: ***********************************************************************************************************************************************************************************************************************************************************************************
19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-3.0.3.xml
19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client: Twitter4J
19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Accept-Encoding: gzip
19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – User-Agent: twitter4j http://twitter4j.org/ /3.0.3
19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-Version: 3.0.3
19:12:37.382 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Connection: close
19:12:37.383 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Post Params: count=0&track=happy&stall_warnings=true
19:12:37.452 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
19:12:37.452 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – Waiting for 16000 milliseconds
19:12:37.452 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Waiting for 16000 milliseconds]
19:12:53.453 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
19:12:53.453 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] INFO twitter4j.TwitterStreamImpl – Establishing connection.
19:12:53.453 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Establishing connection]
19:12:53.454 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Request:
19:12:53.454 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – POST https://stream.twitter.com/1.1/statuses/filter.json
19:12:53.455 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth base string: POST&https%3A%2F%2Fstream.twitter.com%2F1.1%2Fstatuses%2Ffilter.json&count%3D0%26oauth_consumer_key%3DVzdS7zf33IyW8IdFZYhHg%26oauth_nonce%3D-171852571%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1375927973%26oauth_token%3D1594456190-lXS9kTMkl0x7ttPxcr0RwIEjXix9k3g2xaGikiy%26oauth_version%3D1.0%26stall_warnings%3Dtrue%26track%3Dhappy
19:12:53.456 [Twitter Stream consumer-1[Establishing connection]] DEBUG twitter4j.auth.OAuthAuthorization – OAuth signature: qnPDnV9plkCmdzZzoTQa/7EDyWk=
19:12:53.456 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Authorization: *********************************************************************************************************************************************************************************************************************************************************************************
19:12:53.456 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-3.0.3.xml
19:12:53.456 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client: Twitter4J
19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Accept-Encoding: gzip
19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – User-Agent: twitter4j http://twitter4j.org/ /3.0.3
19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – X-Twitter-Client-Version: 3.0.3
19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Connection: close
19:12:53.457 [Twitter Stream consumer-1[Establishing connection]] DEBUG t.internal.http.HttpClientImpl – Post Params: count=0&track=happy&stall_warnings=true
19:12:53.528 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
19:12:53.529 [Twitter Stream consumer-1[Establishing connection]] INFO twitter4j.TwitterStreamImpl – Waiting for 16000 milliseconds
19:12:53.529 [Twitter Stream consumer-1[Waiting for 16000 milliseconds]] DEBUG twitter4j.TwitterStreamImpl – Twitter Stream consumer-1[Waiting for 16000 milliseconds]
Exit your IDE and run it again – seen this error message when you have already made a connection and it didn’t close correctly
Tank you for your fast reply.
Hi DC,
I tried the code and I am getting following result:
[Wed Aug 21 10:58:49 IST 2013]Establishing connection.
[Wed Aug 21 10:58:53 IST 2013]Extensions not allowed in v2 certificate
[Wed Aug 21 10:58:53 IST 2013]Waiting for 250 milliseconds
[Wed Aug 21 10:58:53 IST 2013]Establishing connection.
[Wed Aug 21 10:58:54 IST 2013]Extensions not allowed in v2 certificate
[Wed Aug 21 10:58:54 IST 2013]Waiting for 500 milliseconds
[Wed Aug 21 10:58:55 IST 2013]Establishing connection.
[Wed Aug 21 10:58:55 IST 2013]Extensions not allowed in v2 certificate
[Wed Aug 21 10:58:55 IST 2013]Waiting for 1000 milliseconds
[Wed Aug 21 10:58:56 IST 2013]Establishing connection.
[Wed Aug 21 10:58:57 IST 2013]Extensions not allowed in v2 certificate
[Wed Aug 21 10:58:57 IST 2013]Waiting for 2000 milliseconds
[Wed Aug 21 10:58:59 IST 2013]Establishing connection.
[Wed Aug 21 10:59:00 IST 2013]Extensions not allowed in v2 certificate
[Wed Aug 21 10:59:00 IST 2013]Waiting for 4000 milliseconds
[Wed Aug 21 10:59:04 IST 2013]Establishing connection.
[Wed Aug 21 10:59:05 IST 2013]Extensions not allowed in v2 certificate
[Wed Aug 21 10:59:05 IST 2013]Waiting for 8000 milliseconds
Please Help!
That looks like you are trying to use HTTPS?
Thanks a lot….it helps me
Hi David Crowley,
I wanna to get the tweets from people following with stream instead of keywords. How to do? Thank you.
This might help – http://stackoverflow.com/questions/2943161/get-tweets-of-a-public-twitter-profile
How can we get the latest tweet from public twitter profile using the streaming api? How do i filter content to get the tweets from a specific public twitter profile.
Hi, Thanks a lot for your code.
Any idea on how to get the tweets using # tag?
this code should work with #tags as well as without #hashtags
Yup, it works. Problem was with the #tag i gave to search. At that moment, there were no recent tweets related to that #tag. Hence, thought it doesn’t work with hastags.
Thanks 🙂
Hi David
That’s a great piece of code . I am completely new to this . I tried the code . I wanted to stop the code after a particular interval of time. But unable to do so. Any help is appreciated.
Thanks
You could loop until you have a certain amount of tweets? or you could use time to do it for so many seconds. Depends on what you are trying to do
Hello David,
there is a compilation error with your code:
SimpleStream.java:22: error: is not abstract and does not override abstract method onStallWarning(StallWarning) in StatusListener
StatusListener listener = new StatusListener() {
^
Can it be fixed?
haven’t seen this error – what version of Twitter4j are you using?
I tried 3.0.3, 3.0.4 and 3.0.5, all came with this error. I use javac compiler (java 7 sdk) on CentOS 6.4
Using Java 1.6/JRE7 and 3.03 using eclipse 3.7 working fine – just tried it there.
thanks nice code
is there any way to get tweets from specific user ?
Do you want it for a logged in user? or any user?
Hi. I’m very comfortable with Java and have done some successful testing with Twitter4J from my IDE, but I’m at a loss when searching for resources on how to deploy a Java that could monitor a twitter feed. Can you point me to some useful resources (i.e. where/how to host, deployment how-tos)? Thanks
This might answer your question – http://stackoverflow.com/questions/6546430/how-to-deploy-a-java-web-application-on-live-server
Hello:
Have you found any way to run multiple listeners with just one TwitterStream connection. ?
Trying to creatively get around some of the limitations that Twitter is imposing, such as only one concurrent TwitterStream connection, only 400 keywords via the fq.track(keywords) method, etc.
I tried creating 2 separate listeners in one method but since they are both connected to the same TwitterStream it treated it as if it were two separate connections and disconnects.
Im beginning to work on a multi-threaded approach to see it that has any merit.
Just curious if you have any ideas.
Thanks
I have never tried it but I presume a multi-threaded approach would work – https://dev.twitter.com/discussions/7542
Hey, thanks for your code. When I ran your code am getting the error as
” sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”
Am using twitter4j-3.0.5, NetBeansIDE7.4, https internet connection in Windows7. Could you pls help me to resolve this issue?
Hey David i just copied your code… But i still get this error below… It looks like, i miss some methods in my Jar-Files. Can u check this out and tell me what i do wrong? If it’s juste the Jar-Files… Can send my the right one.
I use :
twitter4j-stream-3.0.3.jar
twitter4j-core-4.0.1.jar
Exception in thread “main” java.lang.NoClassDefFoundError: twitter4j/internal/http/HttpClientWrapperConfiguration
at twitter4j.TwitterStreamFactory.(TwitterStreamFactory.java:40)
at twitter.SimpleStream.main(SimpleStream.java:23)
Caused by: java.lang.ClassNotFoundException: twitter4j.internal.http.HttpClientWrapperConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
… 2 more
At a guess I would say that using twitter4j-stream-3.0.3.jar
twitter4j-core-4.0.1.jar non matching jars would be the problem if you download Twitter4j then the jars should be of the same version?
I guessed it too, but i also tried each at the time… Then i got other issues: it doesn’t recognize some imported sources. Can u send me the Referenced Libraries u used to compile your code? It’ll be nice, if i can get your code running.
thanks.
I have got this problem fixed by using these 2 Jar-Files:
– twitter4j-stream-3.0.3.jar
– twitter4j-core-3.0.5.jar
Now i get some Twitter-API struggles. And i don’t know what wrong with that problem… It looks like my OAuthConsumerKey, OAuthConsumerSecret, … are false. But that not true, then i used them with the Rest-API and they worked very well.
Please help!
[Fri Apr 04 12:51:54 CEST 2014]Establishing connection.
[Fri Apr 04 12:51:57 CEST 2014]401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.
\n\n\nError 401 Unauthorized
HTTP ERROR: 401
Problem accessing ‘/1.1/statuses/filter.json’. Reason:
Unauthorized
[Fri Apr 04 12:51:57 CEST 2014]Waiting for 10000 milliseconds
[Fri Apr 04 12:52:07 CEST 2014]Establishing connection.
[Fri Apr 04 12:52:07 CEST 2014]401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.
\n\n\nError 401 Unauthorized
HTTP ERROR: 401
Problem accessing ‘/1.1/statuses/filter.json’. Reason:
Unauthorized
[Fri Apr 04 12:52:07 CEST 2014]Waiting for 20000 milliseconds
I am getting the below two lines
[Wed Apr 23 14:54:48 IST 2014]Will use class twitter4j.internal.logging.StdOutLoggerFactory as logging factory.
[Wed Apr 23 14:54:48 IST 2014]Will use twitter4j.internal.http.HttpClientImpl as HttpClient implementation.
That seems ok? Does it run after that?
I dont get any tweets
Have you done a manual search on Twitter.com to see there is related tweets to your search?
Thanks for the valuable information!
I want to know we can limit the number of tweets we get by setting the listener. I mean for a given filter, the onStatus method keeps getting called infinitely. Can we like limit the onStatus method being called for a fixed number of times without using System.exit? Thanks!
You could build some kind of Tweet counter and stop the listener after that amount. Have never tried it. You could also use a Timer and only collect for a set time scale.
Respected Sir,
First of all thanks for your code. But I have a problem. When I run the program it shows exceptions like
Exception in thread “main” java.lang.NoClassDefFoundError: SimpleStream (wrong name: com/crowley/simplestream/SimpleStream)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:472)
I used twitter4j. Please help me.
Looking at this line Exception in thread “main” java.lang.NoClassDefFoundError: SimpleStream (wrong name: com/crowley/simplestream/SimpleStream) i wonder is this the problem?
HI
thanks for the code, could you just explain in detail, how to link the twitter4j files to the program
You need to add the twitter4j jars to the project
i want change the keyword after every 600 seconds but i m not able to do it.Whenever i cleanup and shutdown streamfactory after every keyword it says inflator has been closed.
It only able to fetch 1st keyword.
I am getting Estabilishing Connection. Waiting for 16000 milliseconds continously.
How to bypass proxy in Ubantu ?
Please experts share your ideas…..
Thanks for this great tutorial
Hi
This was an awesome tutorial for newbies like me.
I want to store the output Tweet stream to a JSON file,could you please share any code for that?
Thanks in Advance!
Hi
thanks for this great post. I have a little different problem here. I am trying to get all tweets that are streaming and print it in real time using twitter4j. This means that i should use the listener without any filter, but i don t know how to implement this. Can you help me?
Hi, thanks for the codes… But there’re some problems with my output like:
[Thu Oct 16 12:30:39 SGT 2014]Establishing connection.
[Thu Oct 16 12:30:41 SGT 2014]401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.
\n\n\nError 401 Unauthorized
HTTP ERROR: 401
Problem accessing ‘/1.1/statuses/filter.json’. Reason:
Unauthorized
I’ve applied for the keys and tokens yesterday, but I still get a 401. Can you please help me figure it out? Thank you.
The obvious things to check are that the keys are in the right place and that your system clock is correct?
sir i have a problem
19/05/14 14:15:43 INFO twitter4j.TwitterStreamImpl: Establishing connection.
19/05/14 14:15:44 INFO twitter4j.TwitterStreamImpl: https://stream.twitter.com/1/statuses/filter.json
19/05/14 14:15:44 INFO twitter4j.TwitterStreamImpl: Waiting for 16000 milliseconds
this is showing from longer time
please help
can anyone tell me how to add jar files and setting path through command prompt..please give me the clear steps about what to do after downloading twitter4j 4.0.2 API.Thanks!
If you are using Eclipse you can do it in the IDE – http://www.wikihow.com/Add-JARs-to-Project-Build-Paths-in-Eclipse-%28Java%29 – I would normally use method 2.
I use your code, but have this error:
Exception in thread “main” java.lang.NoClassDefFoundError: twitter4j/internal/http/HttpClientWrapperConfiguration
at twitter4j.TwitterStreamFactory.(TwitterStreamFactory.java:40)
at javaapplication3.JavaApplication3.main(JavaApplication3.java:35)
Caused by: java.lang.ClassNotFoundException: twitter4j.internal.http.HttpClientWrapperConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 2 more
Java Result: 1
The line which caused the error is:
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
Thank you very much!
This might be a fix – http://stackoverflow.com/questions/10301789/could-not-find-class-twitter4j-http-accesstoken
or it seems if you are mixing different versions of twitter4j jars – https://groups.google.com/forum/#!topic/twitter4j/PdSwHkFsIRA
How can i fetch the images from timeline.
Hey, great code helped me a lot! I wanted to know that is there any way to insert these incoming tweet stream to directly into a mysql table using jdbc?
Hello DC,
Any thoughts on how to stream tweets with just the geo-location (geo-boxing) and not using the search keywords inside the track method.
Basically want to stream tweets from a particular country
This post should explain it – https://davidcrowley.me/?p=513
How can i save the output as a .txt file and use it rather than seeing in console..I am new to this help me.
Easy to do – https://stackoverflow.com/questions/1994255/how-to-write-console-output-to-a-txt-file the answers here should help
Thank you.But When i run your above program and save it in a file it overwrites the previous tweets.Any Solution for that?? I need to save all the streaming Tweets in a .txt file
Hey, Aravind you need to change the name of txt file every time in program if you don’t want your collected tweets to be overwritten. Whenever you stop the streaming and a bunch of tweets are collected and next time when you start again to collect new set of tweets just change the name of text file to whatever you want to. This happens because in program PrintStream will first find whether the file with given name on given path exists or not. If file is present then it will overwrite the contents, if not then it will make a new txt file and writes new content(here tweets) in newly created text file. Hope this helps.
Yeah it helps…Thank You 🙂
hi,
will you please provide code for collecting geo-tagged tweets
Hi – I have created this post previously – http://wp.me/p1NE9f-8h
Thank you
How to perform & operation????
You need to copy the code into an IDE and use the external Twitter4j jars
hi,
i am using eclipse tool and twitter 4j api to collect the tweets.How to store the collected tweets in excel sheet ad how to apply the sentiment analysis.please do let me know how to do this.
thanks in advance
You wouldn’t save them in an Excel format. Save them in CSV format. Then you can pick an already existing sentiment analysis package and use that. There is a lot of examples and lots of libraries depending on what language you are comfortable with (Python or R maybe)
Sir i did’nt get how to store the data in CSV format please help me. I have used lexicon based approach to calucate polarity of tweets.
Thank you
getting following error
” The method addListener(StatusListener) is undefined for the type TwitterStream ” when i try to run this as java application
What version of Twitter4j are you using?
crowley sir pls help its very urgent i have hadoop 1.2.1 and ubuntu 14.04 flume 1.6 i m trying to get twitter data in hdfs but
previously i got errror while running flume agent “NO SUCH METHOD FOUND ” which i corrected by googling by replacing twitter jar of version 3..*.* with 2.*.*
BUT NOW I M GETTING ERROR “JAVA.LANG.NULLPOINTEREXCEPTION consumer secret is null”
pls i have tried a lot u caN Screenshots are below
ALSO TO TELL U THAT PREVIOUSLY I GOT ERROR COULD NOT LOAD MAIN CLASS SO I UNSET FLUME HOME
21 Mar 2016 07:51:25,851 ERROR [lifecycleSupervisor-1-5] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) – Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } – Exception follows.
java.lang.NullPointerException: consumer secret is null
at twitter4j.TwitterBaseImpl.setOAuthConsumer(TwitterBaseImpl.java:250)
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:129)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
pls help me i have found it everywhere i can find no answer pls i will be shattered i don’t get help this is my twitter.conf
TwitterAgent.sources= Twitter
TwitterAgent.channels= MemChannel
TwitterAgent.sinks=HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels= MemChannel
TwitterAgent.sources.Twitter.consumerSecret=
XXX
TwitterAgent.sources.Twitter.consumerKey= XXX
TwitterAgent.sources.Twitter.accessTokenSecret= XXXX
TwitterAgent.sources.Twitter.accessToken=
XXXX
TwitterAgent.sources.Twitter.keywords= samsung galaxy,samsung phone launch,samsung mobile
TwitterAgent.sinks.HDFS.channel=MemChannel
TwitterAgent.sinks.HDFS.type=hdfs
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:9000/winit/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
TwitterAgent.channels.MemChannel.type=memory
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.channels.MemChannel.transactionCapacity=1000
Please don’t post with your keys shown as these should be private.
if anyone can help me my mail id singhvnt256@gmail.com
Can you send me code to print all tweets on son format;
What must change on this to code to work;
Thanks you very much
In what format? JSON?
yes json! Sorry for my mistake
Doing something like this http://stackoverflow.com/questions/11968905/how-to-get-a-tweet-with-its-full-json-in-twitter4j should work
Thank you very much!
But i have problem with the raw “for (Tweet tweet : result.getTweets())”
and the problem is the “Tweet” i can’t compile it. Why?
Any error message? And if you are using the code from StackOverFlow then you could just ask there?
cannot find symbol
symbol: class Tweet
location: class TwitterApp2
Can use functional operators
And when i press to fix the bug :
Create a local variable “Tweet”
Use functional Operators
But nothing works!
if you can help me to fix this bug and start download data from twitter in json format.
Thank you
i fix it. The problem was the the twitter4j realase. I do some changes and work
Thank you very much for this example, David!
I had to set proxy host and port to make this program work on my machine:
cb.setHttpProxyHost(“proxy”);
cb.setHttpProxyPort(8080);
I had also set VM args and I think that did the trick instead of cb.setHttp… methods:
-Dhttp.proxyPort=8080 -Dhttp.proxyHost=proxy -Dhttps.proxyPort=8080 -Dhttps.proxyHost=proxy
how to store the twitter data after collecting using twitter4j in database.
It depends on how many tweets are coming in – some help here – https://dev.twitter.com/streaming/overview/processing
Trying to store them in a database on the fly is probably not going to work. You probably need to collect, then process as different tasks.
Please let me know how to store the data in csv format.
Thank you sir
how to add the method of geolcalisation and the language to the program
Hi Sadhana,
Try out these lines just above ConfigurationBuilder in the code. I have not tried this in csv format, but still try this. When I was working with this, I stored data in txt file. While storing you might face mapping issues.
FileOutputStream f = new FileOutputStream(“C:\\your_path\\your_file_name.csv”);
System.setOut(new PrintStream(f));
Thank you Nilang
Hi Sir i have a problem ,,
Exception in thread “main” java.lang.RuntimeException: Uncompilable source code – is not abstract and does not override abstract method onStallWarning(twitter4j.StallWarning) in twitter4j.StatusListener
at Simple.main(Simple.java:31)
how o fix it ?
You don’t have the method implemented – and it must be for StatusListener
Hello sir,
I have a question. I was able to configure everything successfully and collect tweets based on hashtags that I provided in my default list of keywords. Now besides that I am also collecting other hashtags that appear in tweets, the ones that are not in my default keyword list, and then I add them to my default list so that I can receive tweets with those hashtags too.
But my problem is I am focusing on specific event (let’s say olympics) and sometimes I get some hashtags that are not really related to olympics, but they were tweeted with some of the hashtags from my default list. Do you have any idea maybe from statistical or algorithmic point, how could I solve this problem, or I mean how can I decide whether specific hashtags should be added to my default list of keywords or not (is it relevant to the event for which I am trying to collect data)?
I know this might not be related to this discussion, but I would appreciate any kind of advice or suggestion.
Thanks in advance.
This is much more of a research question then a coding question. I think looking at your data would help – collect specific tweets with the hashtag. Look at other hashtag use then evolve the system. Spammers seem to use hashtags a lot – but they often use just trending topics so you could examine both using inclusion of hashtags and then removing tweets that have unrelated trending hashtags in them. Again this could be difficult as hashtags could evolve. Semi related is work by Pavan Kapanipathi & Fabrizio Orlandi http://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=2025&context=knoesis using semantic technologies to take tweet content and tie them to external concepts. This might help
Thanks for the blog…sir…
I am getting rate limit exception ….
Could you please tell me how can I avoid this….
Read the Twitter API guidelines – depends on what you want to do. You could do a request (timed) to stop the rate limiting
I am getting the following error :
I have added java-json.jar
[Mon Feb 27 12:14:10 IST 2017]Establishing connection.
Exception in thread “Twitter Stream consumer-1[Establishing connection]” java.lang.NoClassDefFoundError: twitter4j/internal/org/json/JSONException
at twitter4j.TwitterStreamImpl.getFilterStream(TwitterStreamImpl.java:290)
at twitter4j.TwitterStreamImpl$7.getStream(TwitterStreamImpl.java:279)
at twitter4j.TwitterStreamImpl$7.getStream(TwitterStreamImpl.java:277)
at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:427)
Caused by: java.lang.ClassNotFoundException: twitter4j.internal.org.json.JSONException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 4 more
You could possibly be using mismatched versions? Check if your streaming and search versions of twitter4j are the same
I am using different versions of stream and core files .
If I use the latest version 4.0.4 I get errors at .addListener.
All the errors get resolved after adding “twitter4j-core-3.0.4” , “twitter4j-stream-2.2.5” , “twitter-4j”.
But I get the following error now :
[Mon Feb 27 15:57:10 IST 2017]Establishing connection.
[Mon Feb 27 15:57:13 IST 2017]Connection established.
[Mon Feb 27 15:57:13 IST 2017]Receiving status stream.
Exception in thread “Twitter4J Async Dispatcher[0]” java.lang.NoSuchMethodError: twitter4j.json.JSONObjectType.determine(Ltwitter4j/internal/org/json/JSONObject;)Ltwitter4j/json/JSONObjectType;
at twitter4j.AbstractStreamImplementation$1.run(AbstractStreamImplementation.java:100)
at twitter4j.internal.async.ExecuteThread.run(DispatcherImpl.java:116)
I would use all matching jars – as methods/class structures will have changed between 2.2.5 and 3.0.4
Can you tell me a stable version of core and stream jar files ?
From http://twitter4j.org/en/ – it looks like 4.0.4. But I have yet to try this version with my codebase
4.0.4 doesn’t work well.
I tried using 3.0.4 core and stream jar files and it worked well for me .
Getting all the live tweets.
Thanks alot .
It would be really helpful if you would describe how the code works exactly.
I get the following response for content :
RT @farukonalan: 2015’de Daesh’in infaz görüntülerini paylasan Le Pen’in dokunulmazligi “terör propagandasi” kapsaminda kaldiriliyor. https…
Here the URL received is incomplete.This happens alot of times.I wanted to know what is the issue with the data?
Why do I get incomplete tweets ?
Ask Twitter! or the Twitter4j developers – or start looking at the raw JSON data returned by the request.
Thank you.
Also if you look at the original tweet – https://twitter.com/farukonalan/status/836848809426763779 there is no link. So is the link to the images contained in the tweet or maybe the person retweeting it added a link? Then it might have been over the 140 character limit. But a good place to start would be to examine the actual retweet and see what it actually contains
i am getting 404 error…requested resource is not available
Are you online? Using a proxy?
Hello,
I have implemented this and it works like charm. Now if I want to dynamically add more user to follow, how should I go about it? ie The updated list of users to follow gets stored in DB and from there if a new user is added or an old user got removed I need to update my follow predicates and restart the stream in graceful manner. Can you provide a code sample for this
Regards
Rahul
I could – but I am not a contract developer. Probably adding them daily to a list/db and restarting the process when you have updated the list of users you want to follow.
Ty for this tutorial. But what if I want to get all the tweets ever posted on Twitter. Is there a way to do that? I want to analyse all the tweets on Twitter (text mining) that contain a certain keyword (for my project).
it showing from longer time
19/05/14 14:15:43 INFO twitter4j.TwitterStreamImpl: Establishing connection.
19/05/14 14:15:44 INFO twitter4j.TwitterStreamImpl: https://stream.twitter.com/1/statuses/filter.json
19/05/14 14:15:44 INFO twitter4j.TwitterStreamImpl: Waiting for 16000 milliseconds