Full-featured BitTorrent client package and utilities

Overview

torrent

Join the chat at https://gitter.im/anacrolix/torrent GoDoc CircleCI Astronomer Rating GitHub Release PkgGoDev

This repository implements BitTorrent-related packages and command-line utilities in Go. The emphasis is on use as a library from other projects. It's been used 24/7 in production by downstream services since late 2014. The implementation was specifically created to explore Go's concurrency capabilities, and to include the ability to stream data directly from the BitTorrent network. To this end it supports seeking, readaheads and other features exposing torrents and their files with the various Go idiomatic io package interfaces. This is also demonstrated through torrentfs.

There is support for protocol encryption, DHT, PEX, uTP, and various extensions. There are several data storage backends provided: blob, file, bolt, mmap, and sqlite, to name a few. You can write your own to store data for example on S3, or in a database.

Some noteworthy package dependencies that can be used for other purposes include:

Installation

Install the library package with go get github.com/anacrolix/torrent, or the provided cmds with go get github.com/anacrolix/torrent/cmd/....

Library examples

There are some small examples in the package documentation.

Mentions

Downstream projects

There are several web-frontends and Android clients among the known public projects:

Help

Communication about the project is primarily through Gitter and the issue tracker.

Command packages

Here I'll describe what some of the packages in ./cmd do.

Note that the godo command which is invoked in the following examples builds and executes a Go import path, like go run. It's easier to use this convention than to spell out the install/invoke cycle for every single example.

torrent

Downloads torrents from the command-line. This first example does not use godo.

$ go get github.com/anacrolix/torrent/cmd/torrent
# Now 'torrent' should be in $GOPATH/bin, which should be in $PATH.
$ torrent download 'magnet:?xt=urn:btih:KRWPCX3SJUM4IMM4YF5RPHL6ANPYTQPU'
ubuntu-14.04.2-desktop-amd64.iso [===================================================================>]  99% downloading (1.0 GB/1.0 GB)
2015/04/01 02:08:20 main.go:137: downloaded ALL the torrents
$ md5sum ubuntu-14.04.2-desktop-amd64.iso
1b305d585b1918f297164add46784116  ubuntu-14.04.2-desktop-amd64.iso
$ echo such amaze
wow

torrentfs

torrentfs mounts a FUSE filesystem at -mountDir. The contents are the torrents described by the torrent files and magnet links at -metainfoDir. Data for read requests is fetched only as required from the torrent network, and stored at -downloadDir.

$ mkdir mnt torrents
$ godo github.com/anacrolix/torrent/cmd/torrentfs -mountDir=mnt -metainfoDir=torrents &
$ cd torrents
$ wget http://releases.ubuntu.com/14.04.2/ubuntu-14.04.2-desktop-amd64.iso.torrent
$ cd ..
$ ls mnt
ubuntu-14.04.2-desktop-amd64.iso
$ pv mnt/ubuntu-14.04.2-desktop-amd64.iso | md5sum
996MB 0:04:40 [3.55MB/s] [========================================>] 100%
1b305d585b1918f297164add46784116  -

torrent-magnet

Creates a magnet link from a torrent file. Note the extracted trackers, display name, and info hash.

$ godo github.com/anacrolix/torrent/cmd/torrent-magnet < ubuntu-14.04.2-desktop-amd64.iso.torrent
magnet:?xt=urn:btih:546cf15f724d19c4319cc17b179d7e035f89c1f4&dn=ubuntu-14.04.2-desktop-amd64.iso&tr=http%3A%2F%2Ftorrent.ubuntu.com%3A6969%2Fannounce&tr=http%3A%2F%2Fipv6.torrent.ubuntu.com%3A6969%2Fannounce
Comments
  • Slow download speed (Using Elementum)

    Slow download speed (Using Elementum)

    Hi,

    I'm using Elementum 0.0.39, have tried the two options of: File Storage and Memory Storage. On both I got very low speed.

    • File Storage (Max download limit = 8000kbs, Connections limit = 150, Init buffer = 30MB) Speed = ~500kbs - 1200kbs Kodi log: https://paste.ubuntu.com/26496607/ Torrent log: https://paste.ubuntu.com/26496608/

    • Memory Storage (Max download limit = 9000kbs, Connections limit = 150, Init buffer = 100MB, Memory size for use = 250Mb) Speed = ~300kbs - 400kbs Kodi log: https://paste.ubuntu.com/26496618/ Torrent log: https://paste.ubuntu.com/26496622/

    Kodi 17.6 Windows 10, 64 bit @elgatito

    opened by burekas7 153
  • Webtorrent support

    Webtorrent support

    Issuehunt badges

    Does this client by any chance support webtorrent?

    If no, anyone know of a go torrent solution that does?

    thanks!


    IssueHunt Summary

    anacrolix anacrolix has been rewarded.

    Backers (Total: $250.00)

    • steelbrain steelbrain ($50.00)
    • $200.00 have been anonymously funded.

    Submitted pull Requests


    Tips


    IssueHunt has been backed by the following sponsors. Become a sponsor

    enhancement :gift: Rewarded on Issuehunt 
    opened by faddat 59
  • go-libutp build issues on Windows

    go-libutp build issues on Windows

    
    C:\Users\shellus\go\src\github.com\shellus\untitled>go get github.com/anacrolix/torrent
    # github.com/anacrolix/go-libutp
    ..\..\anacrolix\go-libutp\utp.go:12:1: error: unknown type name 'uint64_t'
     uint64_t errorCallback(utp_callback_arguments *);
     ^~~~~~~~
    ..\..\anacrolix\go-libutp\utp.go:13:1: error: unknown type name 'uint64_t'
     uint64_t logCallback(utp_callback_arguments *);
     ^~~~~~~~
    ..\..\anacrolix\go-libutp\utp.go:14:1: error: unknown type name 'uint64_t'
     uint64_t acceptCallback(utp_callback_arguments *);
     ^~~~~~~~
    ..\..\anacrolix\go-libutp\utp.go:15:1: error: unknown type name 'uint64_t'
     uint64_t sendtoCallback(utp_callback_arguments *);
     ^~~~~~~~
    ..\..\anacrolix\go-libutp\utp.go:16:1: error: unknown type name 'uint64_t'
     uint64_t stateChangeCallback(utp_callback_arguments *);
     ^~~~~~~~
    ..\..\anacrolix\go-libutp\utp.go:17:1: error: unknown type name 'uint64_t'
     uint64_t readCallback(utp_callback_arguments *);
     ^~~~~~~~
    
    C:\Users\shellus\go\src\github.com\shellus\untitled>
    
    
    
    opened by shellus 25
  • Good peers getting banned after chunks are removed from storage

    Good peers getting banned after chunks are removed from storage

    I have been testing this library using an in-memory lru cache with a small readahead time. One of the issues I have observed is that when chunks get evicted from the cache and the piece completion state changes, if I then seek back to try to re-download those chunks, in a lot of cases the peers end up getting banned, and the download seems to hang for a long time. I see a bunch of messages like this in the logs:

    2020-01-06 00:50:54 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:07 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:10 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:12 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:18 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:19 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:20 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:21 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:22 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:24 client.go:1222: banning ip 5.39.78.105
    2020-01-06 00:51:26 client.go:1222: banning ip 99.242.55.224
    2020-01-06 00:51:28 client.go:1222: banning ip 185.149.90.102
    2020-01-06 00:51:48 client.go:1222: banning ip 195.154.176.126
    2020-01-06 00:52:25 client.go:1222: banning ip 84.94.62.90
    2020-01-06 00:52:27 client.go:1222: banning ip 185.149.90.32
    2020-01-06 00:53:47 client.go:1222: banning ip 2.44.118.238
    2020-01-06 00:54:09 client.go:1222: banning ip 216.244.83.187
    2020-01-06 00:54:41 client.go:1222: banning ip 195.154.225.54
    

    I mentioned this to you over email and your reply you said:

    My theory is that chunks arriving after piece is marked for hashing are added to the dirty list (after it's already cleared in preparation to be hashed). Later when the pieces are evicted from the cache, a piece check fails and those peers that sent late, overlapping chunks are being banned (despite being innocent). If your cache was small enough, and chunks were redundant, and hashes took a while, this would occur quite a lot. I think the solution is to disallow writing chunks while piece checks are pending or active.

    bug 
    opened by ccampbell 22
  • Hashing is very slow at 16-32 MB/s

    Hashing is very slow at 16-32 MB/s

    Hello,

    I want to use the torrent library in a project that will distribute files across a number of heterogenous storage nodes. I wrote a very simple client, the source code is here: https://gist.github.com/ambientsound/bbc30b258bdd253d5fbaa6ce339e2ff1

    Now, after adding a torrent, the client starts hashing. The hash speed is very slow, however, stuck between 16-32 MB/s. The filesystem can handle around 250 MB/s when doing sha1sum from the command line.

    I expect 10x the performance I can see from the torrent library, and thus assume that I'm doing something wrong. Can you provide some help?

    opened by ambientsound 22
  • Torrent not available after a while when using WebTorrent

    Torrent not available after a while when using WebTorrent

    I used torrent download -seed 'magnet:?xt=urn:btih:...' to download and then seed the torrent via WebTorrent.

    If I try to download the torrent (using instant.io or btorrent.xyz) just after download completes, so after

    downloading "...": 288 MB/288 MB, 2197/2197 pieces completed (0 partial)
    

    is printed to the logs, then everything is alright.

    But then, after waiting for a while I see the following in the logs:

    2020-06-23T11:19:49+0300 INFO  tracker_client.go:100: tracker client for "wss://tracker.btorrent.xyz": websocket instance ended: read message error: websocket: close 1006 (abnormal closure): unexpected EOF
    2020-06-23T11:20:32+0300 INFO  tracker_client.go:100: tracker client for "wss://tracker.openwebtorrent.com": websocket instance ended: read message error: websocket: close 1006 (abnormal closure): unexpected EOF
    

    And immediately after those logs, I am unable to download the seeded file via WebTorrent using the websites mentioned above.

    bug 
    opened by 2m 18
  • Redundancy of loading

    Redundancy of loading

    When viewing a movie in an Elementum or in my program (similar to an Elementum https://github.com/YouROK/TorrServe), downloading useful data (BytesReadUsefulData) is ~20 times smaller than BytesReadData Torrent 7.7 GB In stat: BytesRead ~9.57 GB BytesReadData ~9.55 GB BytesReadUsefulData ~564 MB

    Therefore, the download of the network channel is very large

    Sorry for my english, this is not my native language

    enhancement 
    opened by YouROK 18
  • UDP errors on Windows

    UDP errors on Windows

    When running on Windows, after 1-2 minute of downloading, I get non stop error messages that keep scrolling infinitely. When I restart torrent, it works ok, but after 1 minute the same happens again.

    Screenshot

    When I checked the network activity of the torrent process after the error kicks in, I find out that it "loses" all UDP connections and only TCP connections remain active.

    Screenshot

    This might be an error in UDP socket implementation for Windows in the network libraries you're using. Or something else.

    help wanted windows 
    opened by stunndard 18
  • Why Can't download use custom tracker server?

    Why Can't download use custom tracker server?

    I built a custom tracker server,and create torrent file and create magnt uri,example:

    magnet:?xt=urn:btih:ac3be1065c824d1ff01ecc795dce7b966f5516e2&dn=nginx-1.12.2.tar.gz&tr=http%3A%2F%2F192.168.0.110%3A6969

    so I tried to download it,But I didn't get anything,The code seems to stop at t.Gotinfo ()

    My english is very poor , sorry about that .

    inactive 
    opened by rinyshen 17
  • API cleanup suggestions

    API cleanup suggestions

    as discussed in #218, the API is a little hard to follow and use for certain use cases. The goal here is to discuss and suggest usability improvements to the API. I believe these changes could also be used to help resolve #238, #168 in a clean fashion.

    • remove torrent creation methods from the client. there are just want too many of them. poorly documented, and its unclear which one should be used and when.
    func (cl *Client) AddMagnet(uri string) (T *Torrent, err error) {}
    func (cl *Client) AddTorrentFromFile(filename string) (T *Torrent, err error) {}
    func (cl *Client) AddTorrent(mi *metainfo.MetaInfo) (T *Torrent, err error) {}
    func (cl *Client) AddTorrentInfoHash(infoHash metainfo.Hash) (t *Torrent, new bool) {}
    func (cl *Client) AddTorrentInfoHashWithStorage(infoHash metainfo.Hash, specStorage storage.ClientImpl) (t *Torrent, new bool) {}
    func (cl *Client) AddTorrentSpec(spec *TorrentSpec) (t *Torrent, new bool, err error) {}
    
    • Create new standalone methods for the creation of torrent specifications. these methods have no concept of the client and just know how to create torrent specs.
    type TorrentSpecOption func(*TorrentSpec)
    // TorrentSpecData provide the source data to the torrent, nil can represent no local data.
    type TorrentSpecData(d TorrentImpl) TorrentSpecOption {
      return func(t *TorrentSpec) {
        t.TorrentImpl = d
      }
    }
    
    // New create a torrent from the metainfo.Info and any additional options.
    func New(mi metainfo.Info, options ...TorrentSpecOption) (*TorrentSpec, error) {}
    
    // NewFromData creates a torrent from the raw data the torrent will represent.
    func NewFromData(r io.Reader, options ...TorrentSpecOption) (*TorrentSpec, error) {}
    
    // NewFromFile convience method to create a torrent from a file. see NewFromData for
    // more information.
    func NewFromFile(path string, options ...TorrentSpecOption) (*TorrentSpec, err error) {}
    
    // NewFromMetadata creates a torrent from metadata stored in a file based format.
    func NewFromMetadata(r io.ReadCloser, options ...TorrentSpecOption) (*TorrentSpec, error) {}
    
    // NewFromMagnet create a torrent from a magnet URL.
    func NewFromMagnet(s string, options ...TorrentSpecOption) (*TorrentSpec, error) {}
    
    • Add two new methods to the client that just deal with torrent specs.
    func (cl *Client) AddTorrent(t *TorrentSpec) (new bool, T *Torrent, err error) {}
    func (cl *Client) DropTorrent(t *TorrentSpec) error {}
    
    • possible add a third method to the client to wait for a specific torrent to complete. I'm still undecided on this one, think it might be better on the torrent struct itself.
    func (cl *Client) AwaitCompletion(c context.Context, t *TorrentSpec) (chan error) {}
    

    as an aside, I'm willing to help submit PRs for this work

    enhancement 
    opened by james-lawrence 17
  • Prioritization issue in torrent

    Prioritization issue in torrent

    @anacrolix, we used torrent in our web server to stream high bitrate videos from memory. There was possibility to prioritize further blocks with setPriority and they were downloaded in such order: (A|N)-Actual/Next, (R)-Readahead, (H)-High Priority and (N)-Normal. With latest builds for example H and R blocks loaded faster then N and even A - and there is video streaming issue because A block must be completed before play cursor. Last version TorrServer loaded blocks as expected was probably 1.34.0 (not sure). Is there a way to implement piece load order in current torrent code? https://github.com/YouROK/TorrServer/blob/d4744e2e4751449ba1551bf2c219a62125233d33/server/torr/storage/torrstor/cache.go#L255-L278 https://github.com/YouROK/TorrServer/blob/d4744e2e4751449ba1551bf2c219a62125233d33/server/torr/storage/torrstor/cache.go#L353-L365 L1 L2

    opened by tsynik 16
  • Adding custom authentication headers

    Adding custom authentication headers

    I'm looking at adding custom headers to http or websocket requests that I send from the client to a tracker I control. The idea is to filter the requests coming to the tracker to only allow the ones from clients authenticated via a token.

    It doesn't seem like there's currently an option for this in the client. How would you approach this @anacrolix?

    opened by marcovidonis 6
  • Sounds like that i can't download all of it

    Sounds like that i can't download all of it

    I have got some torrent's file from here:https://archive.org/ But sounds like that all of these torrent can't be downloaded.

    There are some logs below:

    2022-11-06T19:09:57+0800 NIL [torrent 55e5f8c36e7cb3208b164f31d3e0b28b0e687d04 github.com/anacrolix/torrent.(*webseedPeer).requestResultHandler:169]: "alice_in_wonderland_librivox": webseed peer for "/8/items/": Request {0 {49152 16384}} rejected: reading "/8/items/alice_in_wonderland_librivox/AliceInWonderland_librivox.m4b" at "bytes=49152-65535": Get "/8/items/alice_in_wonderland_librivox/AliceInWonderland_librivox.m4b": unsupported protocol scheme ""
    2022-11-06T19:09:57+0800 NIL [github.com/anacrolix/torrent.(*webseedPeer).requestResultHandler:174]: closing webseed peer for "/8/items/"
    2022-11-06T19:09:57+0800 NIL [github.com/anacrolix/torrent.(*webseedPeer).requester.func1:97]: requester 3: error doing webseed request {0 {49152 16384}}: reading "/8/items/alice_in_wonderland_librivox/AliceInWonderland_librivox.m4b" at "bytes=49152-65535": Get "/8/items/alice_in_wonderland_librivox/AliceInWonderland_librivox.m4b": unsupported protocol scheme ""
    

    It shows 'unsupported protocol scheme ""'. I see your internal examples had this case,and this library support webseed. But why all my case from here(https://archive.org/) is failed? BTW, magnet or torrent file from other origins works well.Thanks.

    opened by Shawn-Huang-Tron 1
  • Can't get this repo

    Can't get this repo

    opened by Shawn-Huang-Tron 16
  • Update `crawshaw.io/sqlite` dependencies version

    Update `crawshaw.io/sqlite` dependencies version

    i used gcc version

    $ gcc --version
    gcc (GCC) 12.1.1 20220730
    Copyright © 2022 Free Software Foundation, Inc.
    

    example go code

    package main
    
    import "github.com/anacrolix/torrent"
    
    func main() {
      c, _ := torrent.NewClient(nil)
      defer c.Close()
    
      t, _ := c.AddMagnet("magnet:?xt=urn:btih:ZOCMZQIPFFW7OLLMIC5HUB6BPCSDEOQU")
      <- t.GotInfo()
      t.DownloadAll()
      c.WaitAll()
    }
    

    it's warning when build

    $ go build     
    # crawshaw.io/sqlite
    In file included from ../../go/pkg/mod/crawshaw.io/[email protected]/static.go:19:
    ../../go/pkg/mod/crawshaw.io/[email protected]/./c/sqlite3.c: 在函数‘sqlite3Fts5IndexQuery’中:
    ../../go/pkg/mod/crawshaw.io/[email protected]/./c/sqlite3.c:220863:18: 警告:‘memcpy’ specified bound 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
    220863 |     if( nToken ) memcpy(&buf.p[1], pToken, nToken);
           |
    

    and i found this issue, It's caused by sqlite, crawshaw.io/sqlite has been fixed, please update the crawshaw.io/sqlite dependency version

    opened by fnoopv 1
  • Use consistent receiver names across methods.

    Use consistent receiver names across methods.

    There are a few places in the codebase where different methods of the same object have different receiver names. Sometimes it might be slightly confusing, updating the code to have consistent naming.

    opened by veshij 1
Owner
Matt Joiner
I'm a functional programming and concurrency enthusiast that got sucked into Go. Please send help.
Matt Joiner
🌧 BitTorrent client and library in Go

rain BitTorrent client and library in Go. Running in production at put.io. Features Core protocol Fast extension Magnet links Multiple trackers UDP tr

Cenk Altı 784 Nov 25, 2022
Your Fluffy and Neighbourly Bittorrent Client

Totorrent: A Fluffy and Neighbourly Bittorrent Client This is a WIP bittorrent client I've been working on for a couple of months. I think it's in a v

Abdullah Joseph 3 Jul 28, 2021
BitTorrent client and library in Go. Running in production at put.io.

rain BitTorrent client and library in Go. Running in production at put.io. Integration of embedded gRPC framework (All APIs translated to gRPC Server-

Suhas Aggarwal 2 Apr 11, 2022
Server and client implementation of the grpc go libraries to perform unary, client streaming, server streaming and full duplex RPCs from gRPC go introduction

Description This is an implementation of a gRPC client and server that provides route guidance from gRPC Basics: Go tutorial. It demonstrates how to u

Joram Wambugu 0 Nov 24, 2021
gqlgenc is a fully featured go gql client, powered by codegen

gqlgenc Note: ⚠️ This is a WIP, backward-compatibility cannot be guaranteed yet, use at your own risk gqlgenc is a fully featured go gql client, power

Infiot Inc 29 Sep 17, 2022
torsniff - a sniffer that sniffs torrents from BitTorrent network

torsniff - a sniffer that sniffs torrents from BitTorrent network English | 简体中文 Introduction torsniff is a torrent sniffer, it sniffs torrents that p

null 3.9k Nov 30, 2022
🐈🌐 nyaa.si terminal BitTorrent tracker

koneko - TUI Client for nyaa.si ?? ?? Download Anime, Literature, Audio, Live Action, Pictures, Software Data - go-nyaa TUI - tview Installation ?? go

Ilya Revenko 173 Dec 4, 2022
Validator node for Bittorrent Chain Network

Delivery Validator node for Bittorrent Chain Network. It uses peppermint, customized Tendermint. Install from source Make sure your have go1.11+ alrea

PanoptesDev 1 Dec 14, 2021
P2PDistributedHashTable - A golang Kademlia/Bittorrent DHT library that implements BEP5

This is a golang Kademlia/Bittorrent DHT library that implements BEP 5. It's typ

Suhas Aggarwal 2 Apr 10, 2022
DeSo is a blockchain built from the ground up to support a fully-featured social network

DeSo is a blockchain built from the ground up to support a fully-featured social network. Its architecture is similar to Bitcoin, only it supports complex social network data like profiles, posts, follows, creator coin transactions, and more.

DeSo Protocol 296 Dec 1, 2022
Inspired by go-socks5,This package provides full functionality of socks5 protocol.

The protocol described here is designed to provide a framework for client-server applications in both the TCP and UDP domains to conveniently and securely use the services of a network firewall.

Zhangliu 68 Sep 27, 2022
A Go package for creating contributor list by release, Help full for those organization that use one repository for platform release

This is a Go package which create contributors list by release by scanning across all repository that exist in organisation, Only helpful for those or

Yuvraj 0 Dec 26, 2021
An experimental package that rely on go generics to implement collection functions utilities

go-underscore go-underscore is a utility-belt library for Golang that provides s

null 21 Mar 20, 2022
A C/S Tool to Download Torrent Remotely and Retrieve Files Back Over HTTP at Full Speed without ISP Torrent Limitation.

remote-torrent Download Torrent Remotely and Retrieve Files Over HTTP at Full Speed without ISP Torrent Limitation. This repository is an extension to

Bruce Wang 59 Sep 30, 2022
SOCKS Protocol Version 5 Library in Go. Full TCP/UDP and IPv4/IPv6 support

socks5 中文 SOCKS Protocol Version 5 Library. Full TCP/UDP and IPv4/IPv6 support. Goals: KISS, less is more, small API, code is like the original protoc

TxThinking 499 Dec 5, 2022
Baseledger core consensus for running validator, full and seed nodes

baseledger-core Baseledger core consensus client for running a validator, full or seed node. ⚠️ WARNING: this code has not been audited and is not rea

Baseledger 0 Jan 13, 2022
Wrapper around bufcli to make it do cross-repo compiles for private repos and use full paths.

Bufme A tool for compiling protos with full directory paths and cross repo compiles. Introduction Protocol buffers rock, but protoc should die in a fi

John Doak 0 Feb 5, 2022
EU Digital Covid Certificate utilities in Go [Create, Validate and Parse Green-Pass/EU-DCC]

go-dcc EU Digital Covid Certificate utilities in Go [Create, Validate and Parse Green-Pass/EU-DCC] Repo work in-progress CLI Usage: ######Create and S

Francesco 1 Dec 23, 2021
proxylogon, proxyshell, proxyoracle full chain exploit tool

Proxy-Attackchain proxylogon, proxyshell, proxyoracle full chain exploit tool ProxyLogon: The most well-known and impactful Exchange exploit chain Pro

lUc1f3r11 204 Nov 27, 2022