Scrape the Twitter Frontend API without authentication with Golang.

Overview

Twitter Scraper

Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse-engineered. No API rate limits. No tokens needed. No restrictions. Extremely fast.

You can use this library to get the text of any user's Tweets trivially.

Installation

go get -u github.com/n0madic/twitter-scraper

Usage

Get user tweets

package main

import (
    "context"
    "fmt"
    twitterscraper "github.com/n0madic/twitter-scraper"
)

func main() {
    scraper := twitterscraper.New()
    for tweet := range scraper.GetTweets(context.Background(), "Twitter", 50) {
        if tweet.Error != nil {
            panic(tweet.Error)
        }
        fmt.Println(tweet.Text)
    }
}

It appears you can ask for up to 50 tweets (limit ~3200 tweets).

Search tweets by query standard operators

Tweets containing “twitter” and “scraper” and “data“, filtering out retweets:

package main

import (
    "context"
    "fmt"
    twitterscraper "github.com/n0madic/twitter-scraper"
)

func main() {
    scraper := twitterscraper.New()
    for tweet := range scraper.SearchTweets(context.Background(),
        "twitter scraper data -filter:retweets", 50) {
        if tweet.Error != nil {
            panic(tweet.Error)
        }
        fmt.Println(tweet.Text)
    }
}

The search ends if we have 50 tweets.

See Rules and filtering for build standard queries.

Set search mode

scraper.SetSearchMode(twitterscraper.SearchLatest)

Options:

  • twitterscraper.SearchTop - default mode
  • twitterscraper.SearchLatest - live mode
  • twitterscraper.SearchPhotos - image mode
  • twitterscraper.SearchVideos - video mode

Get profile

package main

import (
    "fmt"
    twitterscraper "github.com/n0madic/twitter-scraper"
)

func main() {
    scraper := twitterscraper.New()
    profile, err := scraper.GetProfile("Twitter")
    if err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", profile)
}

Get trends

package main

import (
    "fmt"
    twitterscraper "github.com/n0madic/twitter-scraper"
)

func main() {
    scraper := twitterscraper.New()
    trends, err := scraper.GetTrends()
    if err != nil {
        panic(err)
    }
    fmt.Println(trends)
}

Use http proxy

err := scraper.SetProxy("http://localhost:3128")
if err != nil {
    panic(err)
}

Load timeline with tweet replies

scraper.WithReplies(true)

Default Scraper (Ad hoc)

In simple cases, you can use the default scraper without creating an object instance

import twitterscraper "github.com/n0madic/twitter-scraper"

// for tweets
twitterscraper.GetTweets(context.Background(), "Twitter", 50)
// for tweets with replies
twitterscraper.WithReplies(true).GetTweets(context.Background(), "Twitter", 50)

// for search
twitterscraper.SearchTweets(context.Background(), "twitter", 50)

// for profile
twitterscraper.GetProfile("Twitter")

// for trends
twitterscraper.GetTrends()
Issues
  • how to connect tweet and its reply?

    how to connect tweet and its reply?

    Getting tweets with replies? but don't understand how to connect a tweet with its reply? Because I did not find any field in the 'Tweet' struct for the same. Please provide a solution.

    opened by wraza2k9 13
  • error in getting username of tweet reply

    error in getting username of tweet reply

    The code "tweet.InReplyToStatus.Username" generates an error while accessing the tweet of type 'reply'.

    Please check and assist to resolve the problem. Thanks.

    opened by wraza2k9 10
  • Not getting userid or name who retweeted

    Not getting userid or name who retweeted

    SearchTweets function only returns the userid or name of the person who creates the tweet. It does not return the username/id of the person who retweets

    Can you provide the example use of Retweet Struct?

    Thanks.

    opened by wraza2k9 9
  • Rate Limit Exceeded

    Rate Limit Exceeded

    During scraping the following error has been observed:

    panic: response status 429 Too Many Requests: {"errors":[{"code":88,"message":"Rate limit exceeded."}]}

    Process exiting with code: 2 signal: false

    opened by wraza2k9 7
  • Multiple query filters

    Multiple query filters

    Love the library. Implementing it in my first Go project right now. I am having the issue where I cannot use multiple filters in a query such as:

    ...SearchTweets(context.Background(), "NASA -filter:retweets within_time:1h",50)

    Now obviously NASA is frequently tweeted about, but if I choose something like JASMY, it gives me tweets from outside the hour. I've used this same exact string literal in the Twitter search bar and it seems to function properly there. Am I missing something from your documentation? It seemed to me that we could use the exact same queries as on Twitter itself.

    opened by jonreesman 6
  • Issues about HTML field

    Issues about HTML field

    The latest version is perfect, i can get more content than before. But at tw.HTML field the regular expression of user name (reUsername) is missing the digit rules, such as @RonaldAraujo939 can't be matched.

    And some emoji that defined by Twitter can't be displayed. I remember that the previous version had image resources of emoji.

    WX20201218-180210 WX20201218-175733

    opened by LarchLiu 6
  • how to find pinned tweets from a specific user?

    how to find pinned tweets from a specific user?

    I've tried GetProfile() and GetTweets() but I can't get the top pinned tweet of a user, which is normally displayed when I access the profile via browser/app.

    With FetchTweets(Username,"") I get 400 Bad Request btw.

    Any hints? :-)

    opened by mipak 5
  • Too many requests error on Heroku

    Too many requests error on Heroku

    I deployed this on Heroku and on any request I get

    response status 429 Too Many Requests
    2021-02-22T15:02:25.438432+00:00 app[web.1]: /tmp/build_89d618b3/main.go:77 (0x8fd6ee)
    2021-02-22T15:02:25.438432+00:00 app[web.1]: /tmp/codon/tmp/cache/go-path/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0x8df4b9)
    2021-02-22T15:02:25.438433+00:00 app[web.1]: /tmp/build_89d618b3/main.go:111 (0x8fe0a2)
    2021-02-22T15:02:25.438433+00:00 app[web.1]: /tmp/codon/tmp/cache/go-path/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0x8df4b9)
    2021-02-22T15:02:25.438433+00:00 app[web.1]: /tmp/codon/tmp/cache/go-path/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:83 (0x8f3349)
    2021-02-22T15:02:25.438434+00:00 app[web.1]: /tmp/codon/tmp/cache/go-path/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0x8df4b9)
    2021-02-22T15:02:25.438434+00:00 app[web.1]: /tmp/codon/tmp/cache/go-path/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:241 (0x8f2470)
    2021-02-22T15:02:25.438434+00:00 app[web.1]: /tmp/codon/tmp/cache/go-path/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0x8df4b9)
    2021-02-22T15:02:25.438435+00:00 app[web.1]: /tmp/codon/tmp/cache/go-path/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:409 (0x8e963c)
    2021-02-22T15:02:25.438435+00:00 app[web.1]: /tmp/codon/tmp/cache/go-path/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:367 (0x8e8d3d)
    2021-02-22T15:02:25.438435+00:00 app[web.1]: /tmp/codon/tmp/cache/go1.12.17/go/src/net/http/server.go:2774 (0x6b7167)
    2021-02-22T15:02:25.438436+00:00 app[web.1]: /tmp/codon/tmp/cache/go1.12.17/go/src/net/http/server.go:1878 (0x6b2d50)
    2021-02-22T15:02:25.438436+00:00 app[web.1]: /tmp/codon/tmp/cache/go1.12.17/go/src/runtime/asm_amd64.s:1337 (0x459a00)
    

    I reduced the number of tweets to 1 and still the same error.

    Is this related to this package or Heroku or Gin?

    opened by strongSoda 4
  • response status: 400 Bad Request

    response status: 400 Bad Request

    Hi,

    Great library, very useful. I think Twitter made a change somewhere. Code which used to work to get tweets from a profile now returns "response status: 400 Bad Request".

    Also other functions such as this one broke :

    package main
    
    import (
        "fmt"
        twitterscraper "github.com/n0madic/twitter-scraper"
    )
    
    func main() {
        profile, err := twitterscraper.GetProfile("twitter")
        if err != nil {
            panic(err)
        }
        fmt.Printf("%+v\n", profile)
    }
    

    Thanks again for the work

    opened by Mathiasme 4
  • Some users are not fetched

    Some users are not fetched

    I'm having an issue with fetching some users; i.e fetching @flockinco doesn't seem to work.

    Many others I've tried do work, but this one specifically doesn't. Anything I can do to debug this?

    Thanks!

    opened by flockin 4
  • Use Advanced search endpoint to avoid rate limits

    Use Advanced search endpoint to avoid rate limits

    Use advanced search to avoid rate limits when fetching tweets from a ton of different accounts.

    One limitation I found is that the query can have a maximum of 654 Chars(with URL encoding) or else It fails. Example query (from:someone) since:2022-4-21

    You can find this request by going to twitter advanced search, pressing search, and finding the request that has adaptive.json in chrome dev tools, then Right-click it -> Copy -> Copy as CURL (bash) and then going to postman pressing Import -> Raw -> Paste Clipboard -> Import

    Img

    User Tweets Endpoint Response Headers

    User

    Advanced Search Endpoint Response Headers

    I Tested sending 200+ Requests per second and no rate limit occurred.

    Advanced

    opened by Nojus0 3
  • How can I get user favourites(following) list

    How can I get user favourites(following) list

    I need to get a list of nicknames to which he is subscribed by the user's nickname. I see that the function is currently not implemented. Do you have any ideas how this can be done?

    opened by OrlovDiga 2
Owner
Nomadic
Nomadic
Twitter - Twitter API with oauth in Golang

twitter: Twitter API with oauth in Golang Installation and Usage Install go get

majiaqi 0 Jan 7, 2022
A console based twitter client for displaying tweets from twitter lists

About I follow a bunch of people who span a bunch of topics and wanted a way to keep track of all the cool stuff they post. I figured there would cert

esell 29 Jul 25, 2022
wtf? paranormal twitter.com activity using Twitter Cards. Migros.tr #DoITYourself

GTKE - go-tweet-kart-ele wtf? paranormal twitter.com activity using Twitter Cards. Migros.tr #DoITYourself Just for fun. Go. # You have go. go install

Jan Hacker 0 Dec 7, 2021
A Golang localhost TLS Server for testing Mutual Authentication (A.K.A Client-Side Authentication)

goMutualAuthServer goMutualAuthServer implements a localhost TLS server in Golang, which can be used to perform Mutual Authentication (A.K.A Client-Si

El Mostafa Idrassi 1 Dec 23, 2021
A Go client library for the Twitter 1.1 API

Anaconda Anaconda is a simple, transparent Go package for accessing version 1.1 of the Twitter API. Successful API queries return native Go structs th

Aditya Mukerjee 1.1k Jul 27, 2022
Go Twitter REST and Streaming API v1.1

go-twitter go-twitter is a Go client library for the Twitter API. Check the usage section or try the examples to see how to access the Twitter API. Fe

Dalton Hubble 1.5k Aug 1, 2022
A REST API microservices-based Twitter Clone server.

Simple API Twitter Clone A REST API microservices-based project to fetch, edit, post, and delete tweets. API documentation The API documentation is bu

null 1 May 13, 2022
Twitter backend - Golang

A Twitter-like Website This repository contains the backend code for the final project of Fall 2020 Internet Engineering course. In this project,

ArmanAminian 68 Jun 24, 2022
Twitter ID Finder For Golang

Twitter ID Finder n文字の全数IDを探せます Usage $ make go build -o main main.go $ ./main Twitter ID Finder Creator: @_m_vt Digits: 2 Target IDs: 100 Really? [

null 0 Dec 12, 2021
twitter clone front-end for Internet Engineering course - fall 99 built by go+echo

twitter backend build twitter-like back-end for internet engeering course - fall 99 with go+echo+gorm team mates Roozbeh Sharifnasab + Parsa Fadaee +

Roozbeh Sharifnasab 16 Mar 1, 2022
Periodically collect data about my Twitter account and check in to github to preserve an audit trail.

Twitter audit trail backup This repository backs up my follower list, following list, blocked accounts list and muted accounts list periodically using

Ahmet Alp Balkan 59 Aug 3, 2022
An app/container built in Go to automate a Twitter account using Notion

Notion Tweeter Notion Tweeter is a utility I built using Go to help automate scheduling my tweets using Notion as a backend. More documentation coming

Brian Morrison II 1 Jan 13, 2022
Rimgo - An alternative frontend for Imgur. Based on rimgu and rewritten in Go

rimgo An alternative frontend for Imgur. Based on rimgu and rewritten in Go. It'

null 1 Feb 15, 2022
⚡️ SharePoint authentication, HTTP client & fluent API wrapper for Go (Golang)

Gosip - SharePoint authentication, HTTP client & fluent API wrapper for Go (Golang) Main features Unattended authentication using different strategies

Andrew Koltyakov 79 Jun 18, 2022
Simple no frills AWS S3 Golang Library using REST with V4 Signing (without AWS Go SDK)

simples3 : Simple no frills AWS S3 Library using REST with V4 Signing Overview SimpleS3 is a golang library for uploading and deleting objects on S3 b

Rohan Verma 89 Jul 31, 2022
efsu is for accessing AWS EFS from your machine without a VPN

efsu: VPN-less access to AWS EFS efsu is for accessing AWS EFS from your machine without a VPN. It achieves this by deploying a Lambda function and sh

Glass Echidna 42 Mar 11, 2022
A compute service that lets you run code without provisioning or managing servers

AWS Lambda for Go Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availab

Ibrahim Dauda 0 Oct 13, 2021
Simple-Weather-API - Simple weather api app created using golang and Open Weather API key

Simple Weather API Simple weather api app created using golang and Open Weather

Siva Prakash 3 Feb 6, 2022