BTFS - The First Scalable Decentralized Storage System - A Foundational Platform for Decentralized Applications

Overview

go-btfs

What is BTFS?

BitTorrent File System (BTFS) is a protocol forked from IPFS that utilizes the TRON network and the BitTorrent Ecosystem for integration with DApps and smart contracts.

Table of Contents

Install

The download and install instructions for BTFS are over at: https://docs.btfs.io/docs/btfs-demo.

System Requirements

BTFS can run on most Linux, macOS, and Windows systems. We recommend running it on a machine with at least 2 GB of RAM (it’ll do fine with only one CPU core), but it should run fine with as little as 1 GB of RAM. On systems with less memory, it may not be completely stable. Only support compiling from source for mac and unix-based system.

Install Pre-Built Packages

We host pre-built binaries at https://github.com/TRON-US/btfs-binary-releases.

Initialize a BTFS Daemon

$ btfs init
initializing BTFS node at /home/ubuntu/.btfs
generating 2048-bit keypair...done
peer identity: 16Uiu2HAmHmW9mHcE9c5UfUomy8caBuEgJVP99MDb4wFtzF8URgzE
to get started, enter:

        btfs cat /btfs/QmPbWqakofrBdDSm4mLUS5RE5QiPQi8JbnK73LgWwQNdbi/readme

Start the Daemon

Start the BTFS Daemon

$ btfs daemon

Build from Source

MacOS

Clone the go-btfs repository

$ git clone https://github.com/TRON-US/go-btfs

Navigate to the go-btfs directory and run make install.

$ cd go-btfs
$ make install

A successful make install outputs something like:

$ make install
go: downloading github.com/tron-us/go-btfs-common v0.2.28
go: extracting github.com/tron-us/go-btfs-common v0.2.28
go: finding github.com/tron-us/go-btfs-common v0.2.28
go version go1.14.1 darwin/amd64
bin/check_go_version 1.14
go install  "-asmflags=all='-trimpath='" "-gcflags=all='-trimpath='" -ldflags="-X "github.com/TRON-US/go-btfs".CurrentCommit=e4848946d" ./cmd/btfs

Afterwards, run btfs init and btfs daemon to initialize and start the daemon.

Linux VM

Developers wishing to run a BTFS daemon on a Linux VM should first set up the environment. On an AWS EC2 Linux machine for example, it would be helpful to first install the following tools and dependencies:

$ sudo yum update          // Installs general updates for Linux
$ sudo yum install git     // Lets you git clone the go-btfs repository
$ sudo yum install patch   // Required for building from source
$ sudo yum install gcc     // Required for building from source

Building BTFS from source requires Go 1.14 or higher. To install from the terminal:

$ cd /tmp
$ GO_PACKAGE=go1.14.linux-amd64.tar.gz
$ wget https://golang.org/dl/$GO_PACKAGE
$ sudo tar -xvf $GO_PACKAGE
$ sudo mv go /usr/local
$ sudo rm $GO_PACKAGE

Navigate back to root directory and set the Go Path in the environment variables:

$ export GOPATH=${HOME}/go
$ export PATH=$PATH:/usr/local/go/bin
$ export PATH=$PATH:$GOPATH/bin
$ export GO111MODULE=on

Verify the Go version is 1.14 or higher:

$ go version

Navigate to the go-btfs directory and run make install.

$ cd go-btfs
$ make install

Afterwards, run btfs init and btfs daemon to initialize and start the daemon. To re-initialize a new pair of keys, you can shut down the daemon first via btfs shutdown. Then run rm -r .btfs and btfs init again.

Docker

Developers also have the option to build a BTFS daemon within a Docker container. After cloning the go-btfs repository, navigate into the go-btfs directory. This is where the Dockerfile is located. Build the docker image:

$ cd go-btfs
$ docker image build -t btfs_docker .   // Builds the docker image and tags "btfs_docker" as the name 

A successful build should have an output like:

Sending build context to Docker daemon  2.789MB
Step 1/37 : FROM golang:1.15
 ---> 4fe257ac564c
Step 2/37 : MAINTAINER TRON-US <[email protected]>
 ---> Using cache
 ---> 02409001f528

...

Step 37/37 : CMD ["daemon", "--migrate=true"]
 ---> Running in 3660f91dce94
Removing intermediate container 3660f91dce94
 ---> b4e1523cf264
Successfully built b4e1523cf264
Successfully tagged btfs_docker:latest

Start the container based on the new image. Starting the container also initializes and starts the BTFS daemon.

$ docker container run --publish 8080:5001 --detach --name btfs1 btfs_docker

The CLI flags are as such:

  • --publish asks Docker to forward traffic incoming on the host’s port 8080, to the container’s port 5001.
  • --detach asks Docker to run this container in the background.
  • --name specifies a name with which you can refer to your container in subsequent commands, in this case btfs1.

Configure cross-origin(CORS) You need to configure cross-origin (CORS) to access the container from the host.

(host) docker exec -it btfs1 /bin/sh // Enter the container's shell

Then configure cross-origin(CORS) with btfs

(container) btfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["http://$IP:$PORT"]'
(container) btfs config --json API.HTTPHeaders.Access-Control-Allow-Methods '["PUT", "GET", "POST"]'

E.g:

(container) btfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["http://localhost:8080"]'
(container) btfs config --json API.HTTPHeaders.Access-Control-Allow-Methods '["PUT", "GET", "POST"]'

Exit the container and restart the container

(container) exit
(host) docker restart btfs1

You can access the container from the host with http://localhost:8080/hostui .

Execute commands within the docker container:

docker exec CONTAINER btfs add --chunker=reed-solomon FILE

Getting Started

Some things to try

Basic proof of 'btfs working' locally:

echo "hello world" > hello
btfs add --chunker=reed-solomon hello
# This should output a hash string that looks something like:
# QmaN4MmXMduZe7Y7XoMKFPuDFunvEZU6DWtBPg3L8kkAuS
btfs cat <that hash>

Usage

  btfs  - Global p2p merkle-dag filesystem.

  btfs [--config=<config> | -c] [--debug | -D] [--help] [-h] [--api=<api>] [--offline] [--cid-base=<base>] [--upgrade-cidv0-in-output] [--encoding=<encoding> | --enc] [--timeout=<timeout>] <command> ...

SUBCOMMANDS
  BASIC COMMANDS
    init          Initialize btfs local configuration
    add <path>    Add a file to BTFS
    cat <ref>     Show BTFS object data
    get <ref>     Download BTFS objects
    ls <ref>      List links from an object
    refs <ref>    List hashes of links from an object

  BTFS COMMANDS
    storage       Manage client and host storage features
    rm            Clean up locally stored files and objects

  DATA STRUCTURE COMMANDS
    block         Interact with raw blocks in the datastore
    object        Interact with raw dag nodes
    files         Interact with objects as if they were a unix filesystem
    dag           Interact with IPLD documents (experimental)
    metadata      Interact with metadata for BTFS files

  ADVANCED COMMANDS
    daemon        Start a long-running daemon process
    mount         Mount an BTFS read-only mount point
    resolve       Resolve any type of name
    name          Publish and resolve BTNS names
    key           Create and list BTNS name keypairs
    dns           Resolve DNS links
    pin           Pin objects to local storage
    repo          Manipulate the BTFS repository
    stats         Various operational stats
    p2p           Libp2p stream mounting
    filestore     Manage the filestore (experimental)

  NETWORK COMMANDS
    id            Show info about BTFS peers
    bootstrap     Add or remove bootstrap peers
    swarm         Manage connections to the p2p network
    dht           Query the DHT for values or peers
    ping          Measure the latency of a connection
    diag          Print diagnostics

  TOOL COMMANDS
    config        Manage configuration
    version       Show btfs version information
    commands      List all available commands
    cid           Convert and discover properties of CIDs
    log           Manage and show logs of running daemon

  Use 'btfs <command> --help' to learn more about each command.

  btfs uses a repository in the local file system. By default, the repo is
  located at ~/.btfs. To change the repo location, set the $BTFS_PATH
  environment variable:

    export BTFS_PATH=/path/to/btfsrepo

Development

Some places to get you started on the codebase:

Development Dependencies

If you make changes to the protocol buffers, you will need to install the protoc compiler.

BTFS Gateway

BTFS Gateway is a free service that allows you to retrieve files from the BTFS network in your browser directly.

How to use BTFS Gateway

License

MIT

Comments
  • BTFS runs then immediately shuts down

    BTFS runs then immediately shuts down

    I downloaded both the 64 and 32 bit Windows pre-built versions and they both open a Terminal window, show the text as described on here, then the window does not stay open - the application exits and I cannot go to the Dashboard.

    opened by oldcityws 28
  • Vault Address

    Vault Address

    Hello, I get a new error with Vault Address,

    Terminal shows: init settlement err: init vault service: vault init: execution reverted: vault alerady created

    Error: init vault service: vault init: execution reverted: vault alerady created

    Please help me. What can I do? Thanks. Peer: 16Uiu2HAmSdmJd3T8dm7JyW1bG22xbsJgHSJLJ3f4DJiMxZ6tH3dV

    opened by chris110110 15
  • docker image: btfs resolve -r /btns/blabla/: could not resolve name

    docker image: btfs resolve -r /btns/blabla/: could not resolve name

    Hello.

    I'm testing the docker image of btfs 2.0. In no way i'm able to successfully visit the webinterface.

    Both webui and hostui give the same error.

    I'm reverse proxying the webinterface with traefik (as i also do with 1.6)

    i set the node up successfully, and gave the address test tokens https://testfaucet.bt.io/#/

    The port forward 4002/tcp+udp has been done correctly. (4001 is in use already)

    Changing user to btfs
    btfs version 2.0.0
    Found BTFS fs-repo at /data/btfs
    Initializing daemon...
    go-btfs version: 2.0.0-4fd04f0
    Repo version: 10
    System version: amd64/linux
    Golang version: go1.16.12
    Repo location: /data/btfs
    Peer identity: 16Uiu2HAkzYNxmpuRNkczDPRNR57hFNCP2Gvk7RjnjTnnjb2hwE29
    the private key of wallet import format is:  142a5763c17b09f6095531364c83bb545f2eb4e36c801c3ed2feb48f6222c9c0
    the address of Bttc format is:  0xa77cb7Bcd2DF62D0b9E7dAA93d4C5f20c4C2e38A
    the address of Tron format is:  TREoFdjTZgiqiF8ZNWSFFwb4YHZsTVwdVZ
    self vault: 0xb09addfb7af4e4501b533cd7a9c0e67ef88244d9
    Swarm listening on /ip4/127.0.0.1/tcp/4002
    Swarm listening on /ip4/172.19.0.59/tcp/4002
    Swarm listening on /p2p-circuit
    Swarm announcing /ip4/127.0.0.1/tcp/4002
    Swarm announcing /ip4/172.19.0.59/tcp/4002
    Swarm announcing /ip4/mywanip/tcp/4002
    API server listening on /ip4/0.0.0.0/tcp/5001
    WebUI: http://0.0.0.0:5001/webui
    HostUI: http://0.0.0.0:5001/hostui
    Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
    Remote API server listening on /ip4/127.0.0.1/tcp/5101
    Daemon is ready
    

    i have done the following to try to solve the problem. run this inside the container:

    btfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["*"]'
    btfs config --json API.HTTPHeaders.Access-Control-Allow-Methods '["PUT","GET","POST"]'
    btfs config profile apply storage-host-testnet
    

    i can't get the webinterface to work :) i'm getting this error: btfs resolve -r /btns/16Uiu2HAmU9ysnuasmdyq1rRePYTwHntmyhZdfC9wm4qCPQMAh9Qq/: could not resolve name

    Is there anything that I'm doing wrong, or, is there anything i can do to help with this issue. Thanks.

    opened by solipsist01 12
  • Cant withdrawn BTT + No online proof

    Cant withdrawn BTT + No online proof

    Hello,

    Since 2.1.3, I can't withdrawn my funds (9M BTT) in one of my node. I hoped that after the upgrade in 2.2.0 it would have works out, but it's still not working.

    When I try to withdraw my 9M BTT to my wallet, it says that the transfer is successful (I tried in GUI + in CLI). But the funds are still stuck on my node, when I show the balance (GUI + CLI) the funds are still there and couldn't be transfered !

    unknown unknown (1) 2022-08-22 10_23_50-87 98 155 157_30204 - Remote Desktop Connection

    The other issue is about the online proof, no online proof has been sent since the upgrade (2-3 weeks ago)... i don't know if it's related to the issue as described above. 2022-08-22 21_42_25-87 98 155 157_30204 - Remote Desktop Connection

    2022-08-22 21_43_18-87 98 155 157_30204 - Remote Desktop Connection

    Thank you very much for your help !

    opened by MyNameIs2StrikesInYourHead 11
  • Error: init vault service: vault init: vault not deployed by factory

    Error: init vault service: vault init: vault not deployed by factory

    C:\Users\BTT\AppData\Roaming\BitTorrent\btfs>btfs daemon --chain-id 199 Initializing daemon... go-btfs version: 2.1.0-56bba50 Repo version: 10 System version: amd64/windows Golang version: go1.15.15 Repo location: E:\btfs_data Peer identity: 16Uiu2HAm9XyGbH21A96s6 the address of Bttc format is: 0xDFE687526Acf28160 the address of Tron format is: TWP5rFB8fUtx4kDw self vault: 0xcc380dd30148cf3 init settlement err: init vault service: vault init: vault not deployed by factory

    Error: init vault service: vault init: vault not deployed by factory

    opened by asd976385560 9
  • New conracts without CHEQUE and and disks spaces

    New conracts without CHEQUE and and disks spaces

    In the dashboard, the number of contracts has increased, but the amount of CHEQUE has not increased, and the EARNING HISTORY chart also does not display new contracts, nor has the disk space occupied increased. There were errors in BTFS.exe, but I reloaded the executable file and now I can’t see these errors, I tried to find the log file but it is not there. According to the document https://docs.btfs.io/docs/mainnet-rpc-configure changed the RPC addresses from https://rpc.bittorrentchain.io to the one recommended in the document https://bttc.trongrid.io/. Hope this helps. My question is where to look at the log of the knife or how to enable logging, this will help me and the developer if I have errors again. OS used Win 2008R2

    opened by Cr-opas 8
  • The node is validated for a very long time  BTFS version 2.1.1

    The node is validated for a very long time BTFS version 2.1.1

    Hi. The node is validated for a very long time BTFS version 2.1.1 more than a week. And nothing happens. Check please. it is mainnet. HOST ID 16Uiu2HAmHAFpum168zPzwMB2TChazMmbtTTi8o27Dbb1GHyvv5Sj

    opened by AlexeiKaDev 8
  • Contracts on 0 after update to 2.1.2

    Contracts on 0 after update to 2.1.2

    Hi,

    I've updated to 2.1.2 on linux, and in the dashboard I see that there are 0 contracts, before updating there were a lot more contracts is this a dashboard glitch/bug?

    Regards, Richard

    opened by rooie1971 7
  • could not resolve name

    could not resolve name

    Daemon is ready but open http://127.0.0.1:5001/hostui error:btfs resolve -r /btns/16Uiu2HAmU9ysnuasmdyq1rRePYTwHntmyhZdfC9wm4qCPQMAh9Qq/: could not resolve name

    opened by wecando168 7
  • Issues on running btfs deamon in cloud

    Issues on running btfs deamon in cloud

    Hi team. I'm experimenting with btfs deployment to a cloud. When run btfs daemon on ubuntu I get next errors from time to time:

    1. init settlement err: init vault service: vault init: 503 Service Temporarily Unavailable
    Error: init vault service: vault init: 503 Service Temporarily Unavailable: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
    
    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
        <head>
            <title>The page is temporarily unavailable</title>
            <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
            <style type="text/css">
                /*<![CDATA[*/
                body {
                    background-color: #fff;
                    color: #000;
    
    1. init settlement err: init swap service error
    2. When at least deamon is up, then I have issues with uploading a file to storage.
    session: 0a5e4ceb-d5ad-47cc-a30f-fdc1a4ae2623 entered state: error, msg: failed to find more valid hosts, please try again later
    

    Can you help?

    opened by SergeySeroshtan 6
  • testnet environment  failed to find more valid hosts, please try again later

    testnet environment failed to find more valid hosts, please try again later

    can not upload file show failed to find more valid hosts, please try again later,I try lots of time but all fail.

    exec below command: btfs init -p storage-host-testnet btfs daemon --chain-id 1029 and valut have wbtt,the address have btt to。 all operation in testnet environment。 peer id 16Uiu2HAmNR3xSySFQAuJ5xcwAnUPJXfqurWrqo9kU4vw6uiCz3jx can you help solve this problem?

    image

    opened by linyintao2010 6
  • Enhancement: Enable auto-redirect from root URL to dashboard

    Enhancement: Enable auto-redirect from root URL to dashboard

    Enables auto-redirect from the root URL (http://127.0.0.1:5001/) to the BTFS dashboard (http://127.0.0.1:5001/btfs/QmW3....GniTT/#/admin/dashboard) , instead of serving a 404 error.

    Initially discussed in Discord that this would help newcomers and allow ease of use, where you don't have to remember any dashboard-specific URL to access it.

    One thing to note for consideration from the team: This may present a security issue, as anyone browsing to the root URL will be automatically redirected to the dashboard. If you are visiting for the first time, you will have to first configure the API endpoint.

    opened by pkyriacou 0
Releases(btfs-v2.2.1.1)
Owner
BitTorrent Inc.
BitTorrent Inc.
A simple re-creation of the first level for Space Invaders using Go and SDL-2.

Space Invaders Go Written in Go using SDL2 Usage This application is a sample first level for a re-creation of Space Invaders. The structure of the pr

Brandon Jernigan 0 Feb 27, 2022
Simple, configurable and scalable Structured Logging for Go.

log Log is a simple, highly configurable, Structured Logging library Why another logging library? There's allot of great stuff out there, but also tho

Go Playgound 283 Sep 26, 2022
Cloudinsight Agent is a system tool that monitors system processes and services, and sends information back to your Cloudinsight account.

Cloudinsight Agent 中文版 README Cloudinsight Agent is written in Go for collecting metrics from the system it's running on, or from other services, and

cloudinsight-backup 368 Nov 3, 2022
System resource usage profiler tool which regularly takes snapshots of the memory and CPU load of one or more running processes so as to dynamically build up a profile of their usage of system resources.

Vegeta is a system resource usage tracking tool built to regularly take snapshots of the memory and CPU load of one or more running processes, so as to dynamically build up a profile of their usage of system resources.

Kartik 8 Jan 16, 2022
EdgeLog is a lightweight log management system, and Agent is a part of EdgeLog system

EdgeLog is a lightweight log management system, and Agent is a part of EdgeLog system. It is installed on host machine and its main duty is to collect host program log statics.

null 3 Oct 10, 2022
Log to cloud object storage for golang. Zap extension.

cos-logger Log to cloud object storage for golang implemented as io.Writer. Use it as a plugin/extension to uber-go/zap logger Configure logger and ad

Prashant Sharma 0 Jan 19, 2022
Every 10 minutes, memory, cpu and storage usage is checked and if they over 80%, sending alert via email.

linux-alert Every 10 minutes, memory, cpu and storage usage is checked and if they over 80%, sending alert via email. Usage Create .env file from .env

Meliksah Cetinkaya 0 Feb 6, 2022
xlog is a logger for net/context aware HTTP applications

⚠️ Check zerolog, the successor of xlog. HTTP Handler Logger xlog is a logger for net/context aware HTTP applications. Unlike most loggers, xlog will

Olivier Poitrey 137 Sep 26, 2022
SigNoz helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool

Monitor your applications and troubleshoot problems in your deployed applications, an open-source alternative to DataDog, New Relic, etc. Documentatio

SigNoz 4.7k Sep 24, 2021
A minimal configuration manager for Go applications.

Confetti A simple config manager for Go applications. Install Use the following: go get -u github.com/shivanshkc/confetti/v2 When to use Confetti Con

Shivansh Kuchchal 1 Dec 7, 2021
Example of trace instrumentation in Golang applications :bar_chart:

go-opentelemetry-example Example of trace instrumentation in Golang applications using the opentelemetry. Requirements/dependencies Docker Docker-comp

Gabriel S. Facina 4 Jun 4, 2022
The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

The open-source platform for monitoring and observability. Grafana allows you to query, visualize, alert on and understand your metrics no matter wher

Grafana Labs 52.7k Dec 2, 2022
CNCF Jaeger, a Distributed Tracing Platform

Jaeger - a Distributed Tracing System Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing platform created by Uber Technologies and do

Jaeger - Distributed Tracing Platform 16.8k Dec 8, 2022
Detecctor is a ⚡ fast, fully customizable 💗 monitoring platform. It uses Telegram as a notification 📥 service

Detecctor is a ⚡ fast, fully customizable ?? monitoring platform. It uses Telegram as a notification ?? service. The main components are a TCP server, MongoDB and multiple clients.

null 2 Nov 16, 2021
Open Source Software monitoring platform tools.

ByteOpen Open Source Software monitoring platform tools. Usage Clone the repo to your own go src path cd ~/go/src git clone https://code.byted.org/inf

Ye Xia 2 Nov 21, 2021
Butler - Aggregation and Alerting Platform

Welcome to Butler Table of Contents Welcome About The Project Contributing Developer Workflow Getting Started Configuration About The Project Contribu

Butler 2 Mar 1, 2022
The Prometheus monitoring system and time series database.

Prometheus Visit prometheus.io for the full documentation, examples and guides. Prometheus, a Cloud Native Computing Foundation project, is a systems

Prometheus 45.7k Nov 30, 2022
rtop is an interactive, remote system monitoring tool based on SSH

rtop rtop is a remote system monitor. It connects over SSH to a remote system and displays vital system metrics (CPU, disk, memory, network). No speci

RapidLoop 2k Nov 26, 2022
distributed monitoring system

OWL OWL 是由国内领先的第三方数据智能服务商 TalkingData 开源的一款企业级分布式监控告警系统,目前由 Tech Operation Team 持续开发更新维护。 OWL 后台组件全部使用 Go 语言开发,Go 语言是 Google 开发的一种静态强类型、编译型、并发型,并具有垃圾回

null 824 Nov 21, 2022