Miles to go ... - Arun Gupta: Analyze Donald Trump Tweets with Couchbase and N1Q ...

AWS Serverless Lambda Scheduled Events to Store Tweets in Couchbase explained how to store tweetsin Couchbase using AWS Serverless Lambda. Now, this Lambda Function has been running for a few days and has collected269 tweets from @realDonaldTrump . This blog , inspired by SQL on Twitter: Analysis Made Easy Using N1QL , will show how these tweets can be analyzed usingN1QL.

N1QL is a SQL-likequery language from Couchbase that operates on JSON documents. N1QL and SQL Differences provide differences between N1QL and SQL. Let’s use N1QL toreveal some interesting information from @realDonaldTrump ‘s tweets.

Many thanks to Sitaram from N1QL team to help hack the queries.

HowMany Tweets

First query is to find out how many tweets are available in the database. The query is pretty simple:

Query: SELECTCOUNT(*) tweet_count FROMtwitter;

As you notice, the syntax is very similar to SQL. SELECT , COUNT and FROM clauses are what you are already familiar with from SQL syntax. tweet_count is an alias defined for the returned result. twitter is the bucket where all the JSON documents are stored.

Results: [ { "tweet_count": 269 } ]

The result is a JSON document as well.

Tweet Sample JSON Document

In order to write queries on a JSON document, you need to know the structure of thedocument. The next query will give you that.

Query: SELECT * FROMtwitter LIMIT 1;

The new clause introduced here is LIMIT . This allows to restrict the number of objects that are returned in a result set of SELECT .

Results: [ { "twitter": { "accessLevel": "0", "contributors": [], "createdAt": "1480828438000", "currentUserRetweetId": "-1", "displayTextRangeEnd": "-1", "displayTextRangeStart": "-1", "favoriteCount": "116356", "favorited": false, "geoLocation": null, "hashtagEntities": [], "id": "805278955150471168", "inReplyToScreenName": null, "inReplyToStatusId": "-1", "inReplyToUserId": "-1", "lang": "en", "mediaEntities": [], "place": null, "possiblySensitive": false, "quotedStatus": null, "quotedStatusId": "-1", "rateLimitStatus": null, "retweet": false, "retweetCount": "28330", "retweeted": false, "retweetedByMe": false, "retweetedStatus": null, "scopes": null, "source": "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>", "symbolEntities": [], "text": "Just tried watching Saturday Night Live - unwatchable! Totally biased, not funny and the Baldwin impersonation just can't get any worse. Sad", "truncated": false, "urlentities": [], "user": { "accessLevel": "0", "biggerProfileImageURL": "http://pbs.twimg.com/profile_images/1980294624/DJT_Headshot_V2_bigger.jpg", "biggerProfileImageURLHttps": "https://pbs.twimg.com/profile_images/1980294624/DJT_Headshot_V2_bigger.jpg", "contributorsEnabled": false, "createdAt": "1237383998000", "defaultProfile": false, "defaultProfileImage": false, "description": "President-elect of the United States", "descriptionURLEntities": [], "email": null, "favouritesCount": "46", "followRequestSent": false, "followersCount": "19294404", "friendsCount": "42", "geoEnabled": true, "id": "25073877", "lang": "en", "listedCount": "52499", "location": "New York, NY", "miniProfileImageURL": "http://pbs.twimg.com/profile_images/1980294624/DJT_Headshot_V2_mini.jpg", "miniProfileImageURLHttps": "https://pbs.twimg.com/profile_images/1980294624/DJT_Headshot_V2_mini.jpg", "name": "Donald J. Trump", "originalProfileImageURL": "http://pbs.twimg.com/profile_images/1980294624/DJT_Headshot_V2.jpg", "originalProfileImageURLHttps": "https://pbs.twimg.com/profile_images/1980294624/DJT_Headshot_V2.jpg", "profileBackgroundColor": "6D5C18", "profileBackgroundImageURL": "http://pbs.twimg.com/profile_background_images/530021613/trump_scotland__43_of_70_cc.jpg", "profileBackgroundImageUrlHttps": "https://pbs.twimg.com/profile_background_images/530021613/trump_scotland__43_of_70_cc.jpg", "profileBackgroundTiled": true, "profileBannerIPadRetinaURL": "https://pbs.twimg.com/profile_banners/25073877/1479776952/ipad_retina", "profileBannerIPadURL": "https://pbs.twimg.com/profile_banners/25073877/1479776952/ipad", "profileBannerMobileRetinaURL": "https://pbs.twimg.com/profile_banners/25073877/1479776952/mobile_retina", "profileBannerMobileURL": "https://pbs.twimg.com/profile_banners/25073877/1479776952/mobile", "profileBannerRetinaURL": "https://pbs.twimg.com/profile_banners/25073877/1479776952/web_retina", "profileBannerURL": "https://pbs.twimg.com/profile_banners/25073877/1479776952/web", "profileImageURL": "http://pbs.twimg.com/profile_images/1980294624/DJT_Headshot_V2_normal.jpg", "profileImageURLHttps": "https://pbs.twimg.com/profile_images/1980294624/DJT_Headshot_V2_normal.jpg", "profileLinkColor": "0D5B73", "profileSidebarBorderColor": "BDDCAD", "profileSidebarFillColor": "C5CEC0", "profileTextColor": "333333", "profileUseBackgroundImage": true, "protected": false, "rateLimitStatus": null, "screenName": "realDonaldTrump", "showAllInlineMedia": false, "status": null, "statusesCount": "34269", "timeZone": "Eastern Time (US & Canada)", "translator": false, "url": "https://t.co/mZB2hymxC9", "urlentity": { "displayURL": "https://t.co/mZB2hymxC9", "end": "23", "expandedURL": "https://t.co/mZB2hymxC9", "start": "0", "text": "https://t.co/mZB2hymxC9", "url": "https://t.co/mZB2hymxC9" }, "utcOffset": "-18000", "verified": true, "withheldInCountries": null }, "userMentionEntities": [], "withheldInCountries": null } } ] Top 5 Tweeting Days

After the basic queries are out of the way, let’s look at some interesting data now.

What are the top 5 dayson which @realDonaldTrump tweeted and the tweet count?

Query: SELECTSUBSTR(MILLIS_TO_STR(TO_NUM(cre