đź”—Twitter

đź”—Description

Samples tweets in real-time. Output records will have this schema:

+================================+
| field name  | type             |
+================================+
| id          | long             |
| message     | string           |
| lang        | nullable string  |
| time        | nullable long    |
| favCount    | int              |
| rtCount     | int              |
| source      | nullable string  |
| geoLat      | nullable double  |
| geoLong     | nullable double  |
| isRetweet   | boolean          |
+================================+

đź”—Use Case

The source is used whenever you want to sample tweets from Twitter in real-time. For example, you may want to read tweets and store them in a table where they can be accessed by your data scientists to perform experiments.

đź”—Properties

referenceName: This will be used to uniquely identify this source for lineage, annotating metadata, etc.

See the Twitter OAuth documentation for more information on obtaining your access token and access token secret. The consumer key and secret are specific to your Twitter app. Login, view your apps, then click on the relevant app to find the consumer key and secret.

ConsumerKey: Twitter Consumer Key. (Macro-enabled)

ConsumerSecret: Twitter Consumer Secret. (Macro-enabled)

AccessToken: Twitter Access Token. (Macro-enabled)

AccessTokenSecret: Twitter Access Token Secret. (Macro-enabled)

đź”—Example

{
    "name": "Twitter",
    "type": "realtimesource",
    "properties": {
        "AccessToken": "GetAccessTokenFromTwitter",
        "AccessTokenSecret": "GetAccessTokenSecretFromTwitter",
        "ConsumerKey": "GetConsumerKeyFromTwitter",
        "ConsumerSecret": "GetConsumerSecretFromTwitter"
    }
}

  • CDAP Pipelines Plugin Type: realtimesource
  • CDAP Pipelines Version: 1.6.2-SNAPSHOT