A minimal filecoin client library

Related tags

Network filclient
Overview

filclient

A standalone client library for interacting with the filecoin storage network

Features

  • Make storage deals with miners
    • Query storage ask prices
    • Construct and sign deal proposals
    • Send deal proposals over the network to miners
    • Send data to miners via data transfer and graphsync
    • Check status of a deal with a miner
  • Make retrieval deals with miners
    • Query retrieval ask prices
    • Run retrieval protocol, get data
  • Data transfer management
    • Ability to query data transfer (in or out) status
  • Market funds management
    • Check and add to market escrow balance
  • Local message signing capabilities and wallet management

Roadmap

  • Cleanup and organization of functions
  • Godocs on main methods
  • Direct mempool integration (to avoid relying on a lotus gateway node)
  • Cleanup of dependency tree
    • Remove dependency on filecoin-ffi
    • Remove dependency on go-fil-markets
    • Remove dependency on main lotus repo
  • Good usage examples
    • Sample application using filclient
  • Integration testing

License

MIT

Issues
  • boost integration

    boost integration

    • [x] Test "deals completed" datastore persists between estuary restarts.
    • [x] Test libp2p server manager. In particular, test that Failed event is fired when auth token expires.
    opened by dirkmc 9
  • Feat/subselector retrieval

    Feat/subselector retrieval

    This is an amalgam of https://github.com/filecoin-project/lotus/pull/6393 and the abandoned https://github.com/application-research/estuary/pull/5, providing a fully featured selector retrieval to filc

    Given a random cid status like this, one should be able to:

    filc retrieve -m {{chosenminer}} --datamodel-path-selector Links/217/Hash/Links/30/Hash/Links/0/Hash bafybeihco4vemvoyuca7vqomert5fuvf6deksmakftu654mipagdkehzyu
    opened by ribasushi 7
  • Cannot build Filclient Error : cannot use t.gsReqQueuedHook

    Cannot build Filclient Error : cannot use t.gsReqQueuedHook

    Hi,

    I cannot build the last master, get the following error :

    github.com/filecoin-project/go-data-transfer/transport/graphsync

    /opt/lotus/go/pkg/mod/github.com/filecoin-project/[email protected]/transport/graphsync/graphsync.go:362:88: cannot use t.gsReqQueuedHook (type func(peer.ID, graphsync.RequestData)) as type graphsync.OnIncomingRequestQueuedHook in argument to t.gs.RegisterIncomingRequestQueuedHook

    Complete log :

    DM [email protected]:/usr/src/filclient$ make

    go build
    go: downloading github.com/filecoin-project/lotus v1.13.2-0.20211214230829-0e2278cc76d0
    go: downloading github.com/filecoin-project/go-data-transfer v1.12.0
    go: downloading github.com/filecoin-project/go-fil-markets v1.13.5
    go: downloading github.com/ipfs/go-datastore v0.5.1
    go: downloading github.com/ipfs/go-ipfs-blockstore v1.1.2
    go: downloading github.com/ipfs/go-graphsync v0.11.4
    go: downloading github.com/filecoin-project/go-state-types v0.1.1
    go: downloading go.opentelemetry.io/otel v1.2.0
    go: downloading github.com/ipld/go-car v0.3.3
    go: downloading github.com/libp2p/go-libp2p-core v0.13.0
    go: downloading github.com/ipfs/go-log/v2 v2.4.0
    go: downloading github.com/filecoin-project/go-padreader v0.0.1
    go: downloading github.com/ipfs/go-ipld-cbor v0.0.6
    go: downloading github.com/multiformats/go-multihash v0.1.0
    go: downloading github.com/filecoin-project/specs-actors/v2 v2.3.6
    go: downloading github.com/multiformats/go-base32 v0.0.4
    go: downloading go.opentelemetry.io/otel/trace v1.2.0
    go: downloading github.com/mattn/go-isatty v0.0.14
    go: downloading github.com/ipfs/go-ipfs-ds-help v1.1.0
    go: downloading github.com/ipfs/go-merkledag v0.5.1
    go: downloading github.com/ipld/go-ipld-prime v0.14.3
    go: downloading golang.org/x/sys v0.0.0-20211209171907-798191bca915
    go: downloading golang.org/x/crypto v0.0.0-20211209193657-4570a0811e8b
    go: downloading lukechampine.com/blake3 v1.1.7
    go: downloading github.com/ipfs/go-peertaskqueue v0.7.1
    go: downloading github.com/libp2p/go-msgio v0.1.0
    go: downloading github.com/filecoin-project/go-ds-versioning v0.1.1
    go: downloading github.com/ipfs/go-blockservice v0.2.1
    go: downloading github.com/ipfs/go-ipld-legacy v0.1.1
    go: downloading github.com/filecoin-project/go-statestore v0.2.0
    go: downloading github.com/benbjohnson/clock v1.2.0
    go: downloading github.com/ipfs/go-ipfs-exchange-interface v0.1.0
    go: downloading github.com/filecoin-project/go-crypto v0.0.1
    go: downloading github.com/filecoin-project/go-jsonrpc v0.1.5
    go: downloading github.com/libp2p/go-libp2p-pubsub v0.6.0
    go: downloading github.com/ipld/go-ipld-selector-text-lite v0.0.1
    go: downloading github.com/BurntSushi/toml v0.4.1
    go: downloading github.com/ipfs/go-ds-leveldb v0.5.0
    go: downloading github.com/dgraph-io/badger/v2 v2.2007.3
    go: downloading github.com/ipfs/go-ds-measure v0.2.0
    go: downloading github.com/ipfs/go-ds-badger2 v0.1.2
    go: downloading github.com/gbrlsnchs/jwt/v3 v3.0.1
    go: downloading google.golang.org/genproto v0.0.0-20210917145530-b395a37504d4
    go: downloading github.com/filecoin-project/dagstore v0.4.4
    go: downloading github.com/elastic/go-sysinfo v1.7.0
    go: downloading github.com/go-kit/log v0.2.0
    go: downloading golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2
    go: downloading github.com/cilium/ebpf v0.2.0
    go: downloading github.com/sirupsen/logrus v1.8.1
    go: downloading github.com/coreos/go-systemd/v22 v22.3.2
    go: downloading github.com/cespare/xxhash/v2 v2.1.2
    go: downloading github.com/godbus/dbus/v5 v5.0.4
    go: downloading github.com/go-logfmt/logfmt v0.5.1
    go: downloading github.com/golang/snappy v0.0.3
    go: downloading github.com/libp2p/go-libp2p-discovery v0.6.0
    go: downloading github.com/dgraph-io/ristretto v0.1.0
    go: downloading github.com/libp2p/go-libp2p-peerstore v0.6.0
    go: downloading github.com/golang/glog v1.0.0
    # github.com/filecoin-project/go-data-transfer/transport/graphsync
    /opt/lotus/go/pkg/mod/github.com/filecoin-project/[email protected]/transport/graphsync/graphsync.go:362:88: cannot use t.gsReqQueuedHook (type func(peer.ID, graphsync.RequestData)) as type graphsync.OnIncomingRequestQueuedHook in argument to t.gs.RegisterIncomingRequestQueuedHook
    make: *** [Makefile:66: filclient] Error 2```
    Bug Stale 
    opened by s0nik42 3
  • RetrieveContent failed

    RetrieveContent failed

    My version of filclient: github.com/application-research/filclient v0.0.0-20220504204501-ca86574e3ee5

    I store the file to miner successfully, then retrieve the file using filclient RetrieveContent, the first retrieval successfully retrieves the file, but more calls fail to retrieve it. I think the method blocks at the location in screenshot 1 and cannot return the data. The logs are shown in screenshot 2

    111 222

    opened by leotons 2
  • Should do normal stdout printing for general command output, and logging library should be silent by default

    Should do normal stdout printing for general command output, and logging library should be silent by default

    Using logging library for default command output is presenting a lot of problems. For one, it has to be either enabled manually through env, or enabled automatically in the cli code itself (which overrides env and makes logger unconfigurable). Also, most users probably do not need timestamps. If timestamps are required, that should be handled by filc itself.

    New Feature 
    opened by elijaharita 1
  • Update graphsync with minor fix

    Update graphsync with minor fix

    Goals

    v0.10.8 broke request counts on the Stats method, this fixes it. Also v0.10.9 release happens to also have tracing for responses, which should generate interesting data.

    Updating in Estuary itself after merge is left as an exercise for the reader.

    opened by hannahhoward 1
  • FILC cancels retrievals while a download with blocks already cached is catching up

    FILC cancels retrievals while a download with blocks already cached is catching up

    FILC automatically tries a different miner for retrievals if a download stalls for too long.

    In the case that a download was started, stopped, and then restarted, many blocks will already be present in the blockstore and will be reported as retrieved very quickly. Meanwhile, the download will progress (generally a lot more slowly) also starting from the root node, and since the first downloaded blocks were already marked as retrieved, no progress will be shown for a while after that initial burst. This ends up appearing as a timeout, even though an active download is in fact running.

    One way of fixing this could be to only report explicitly network-retrieved blocks to the retrieval progress callback.

    Bug 
    opened by elijaharita 1
  • Correct traversal options in SelectiveCar calls

    Correct traversal options in SelectiveCar calls

    Without this we could and do exhaust the budget in "diamond-rich" DAGs

    This should fix the following inconsistency:

    ~$ ipfs dag stat bafybeidtrl2wlaxotxavwowzdpnau7ftki2n4z6kebifmerowyijq5wosi
    Size: 32559225580, NumBlocks: 158603
    
    failed to compute commP for bafybeidtrl2wlaxotxavwowzdpnau7ftki2n4z6kebifmerowyijq5wosi: traversal budget exceeded: budget for links reached zero while on path "Links/44/Hash/Links/3/Hash/Links/0/Hash/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/next/0/refs/0/refs/0/refs/1/refs/0/refs/0/next/0/refs/0/next/0/next/0/next/0/next/0/refs/0/next/0/next/0/refs/0/refs/0/next/0" (link: "bafyreifafjpnxvcxrgvip6whbvukbwhdc5y75uldxfhagfpnton357ygdy")
    
    opened by ribasushi 1
  • IPFS retrieval support and automatic miner selection

    IPFS retrieval support and automatic miner selection

    allows filc to retrieve from ipfs first, before trying filecoin. also includes automatic miner selection if none are passed in with the --miner flag, using the estuary /retrieval-candidates endpoint. these are not configurable just yet, but i intend to get to that after a needed refactor/reorganization of filc.

    edit - also includes some simple logic to find most ideal miners based on seal status, cost, and size

    opened by elijaharita 1
  • Move filecoin retrieval progress logging out of filclient

    Move filecoin retrieval progress logging out of filclient

    This information should be provided through a channel or callback, and it should be up to the user of the library to print out this information if desired.

    There is a flag in the FilClient struct relevant to this that will need to be removed.

    On this topic, we should write a small abstraction for printing byte retrieval progress, since this process needs to be done for both FIL and IPFS downloads. It'd be nice to have rate limiting progress printing as well, so terminal writes don't possibly try to bottleneck someone's epic insane download speed.

    New Feature 
    opened by elijaharita 1
  • Fix incorrect saving of multi-block dags, cleanup code

    Fix incorrect saving of multi-block dags, cleanup code

    @elijaharita this fixes an assumption that all data will fit in one block, and bumps some deps prepping for the selector retrieval port ( separate PR )

    opened by ribasushi 1
  • feat: add retrieval state to retrieval events

    feat: add retrieval state to retrieval events

    Adds retrieval event state to the subscribable retrieval events.

    It was a little difficult to add state to the events because there isn't really a shared context between all the different retrieval functions. It would be a bit better if the idea of a Retrieval was a separate context that was created, encapsulating the idea of a single retrieval. Then it can hold state context. I felt that was too big of a lift for right now, but talk of separating storage and retrieval functionality in the future could include this work.

    opened by kylehuntsman 0
  • `doRpc` occasionally hangs and does not respect context cancellation

    `doRpc` occasionally hangs and does not respect context cancellation

    doRpc sometimes hangs temporarily, followed by a stream reset several seconds later. during this time, context cancellations do nothing.

    doRpc takes a context but all it does with it is copy the deadline to the inet stream, so a hanging read or write has no way to close out when the context is canceled.

    a quick fix for this would be to start a goroutine that sets the stream deadline to zero if context cancellation is detected, but not sure how much overhead that'll introduce

    Bug Priority: Low 
    opened by elijaharita 0
  • Missing the ability to do offline deals

    Missing the ability to do offline deals

    Currently when I try filc deal, the option is limited.

    OPTIONS:
       --miner value, -m value  
       --verified               (default: false)
       --announce value         the public multi-address from which to download the data (for deals with protocol v120)
       --help, -h               show help (default: false)
    

    I'd like to see it can do fil+ verified deals to both lotus and boost market nodes. That means more options are needed, such as commP, dataCid, price etc, and the ability to switch between wallet addresses containing the datacap.

    opened by kernelogic 1
  • Data programs version of Filclient (forked)

    Data programs version of Filclient (forked)

    @dkkapur needs the following from filc:

    • track retrieval failures to a central database
      • client address
      • piece CID
      • payload CID
      • timestamp of retrieval finished
      • storage provider ID
      • success/fail
        • if fail, reason
    • this will be in a fork of filclient, no merge master - so as to not increase scope of filclient
    opened by hannahhoward 2
Owner
Application Research Group
Application Research Group
A Filecoin Network sidecar for miners to bid in storage deal auctions.

bidbot Bidbot is a Filecoin Network sidecar for miners to bid in storage deal auctions. Join us on our public Slack channel for news, discussions, and

textile.io 28 Jun 15, 2022
A minimal analytics package to start collecting traffic data without client dependencies.

go-web-analytics A minimal analytics package to start collecting traffic data without client dependencies. Logging incoming requests import "github.co

Jake Kalstad 0 Nov 23, 2021
Tscert - Minimal package for just the HTTPS cert fetching part of the Tailscale client API

tscert This is a stripped down version of the tailscale.com/client/tailscale Go

Tailscale 13 May 31, 2022
Minimal and idiomatic WebSocket library for Go

websocket websocket is a minimal and idiomatic WebSocket library for Go. Install go get nhooyr.io/websocket Highlights Minimal and idiomatic API First

Anmol Sethi 2.3k Jun 24, 2022
Designed to support DNS brute-forcing with a minimal number of network connections

Fast Use of DNS Resolvers Designed to support DNS brute-forcing with a minimal number of network connections. Installation go get -v -u github.com/caf

Jeff Foley 28 Jun 24, 2022
A minimal IPFS replacement for P2P IPLD apps

IPFS-Nucleus IPFS-Nucleus is a minimal block daemon for IPLD based services. You could call it an IPLDaemon. It implements the following http api call

Peergos 19 Jun 12, 2022
Minimal go implementation of a Webauthn L2 relying party

Webauthn-minimial Webauthn-minimal is an implementation of a Webauthn L2 Relying Party using only the go standard library. It uses the utility functon

Arian van Putten 3 Nov 10, 2021
🐉 Simple WireGuard proxy with minimal overhead for WireGuard traffic.

swgp-go ?? Simple WireGuard proxy with minimal overhead for WireGuard traffic. Proxy Modes 1. Zero overhead Simply AES encrypt the first 16 bytes of a

null 23 Jun 27, 2022
Prisma Client Go is an auto-generated and fully type-safe database client

Prisma Client Go Typesafe database access for Go Quickstart • Website • Docs • API reference • Blog • Slack • Twitter Prisma Client Go is an auto-gene

Prisma 1.2k Jul 2, 2022
The Dual-Stack Dynamic DNS client, the world's first dynamic DNS client built for IPv6.

dsddns DsDDNS is the Dual-Stack Dynamic DNS client. A dynamic DNS client keeps your DNS records in sync with the IP addresses associated with your hom

Ryan Young 10 Jun 20, 2022
Go Substrate RPC Client (GSRPC)Go Substrate RPC Client (GSRPC)

Go Substrate RPC Client (GSRPC) Substrate RPC client in Go. It provides APIs and types around Polkadot and any Substrate-based chain RPC calls. This c

Chino Chang 1 Nov 11, 2021
Server and client implementation of the grpc go libraries to perform unary, client streaming, server streaming and full duplex RPCs from gRPC go introduction

Description This is an implementation of a gRPC client and server that provides route guidance from gRPC Basics: Go tutorial. It demonstrates how to u

Joram Wambugu 0 Nov 24, 2021
Godaddy-domains-client-go - Godaddy domains api Client golang - Write automaticly from swagger codegen

Go API client for swagger Overview This API client was generated by the swagger-codegen project. By using the swagger-spec from a remote server, you c

Mickael Stanislas 0 Jan 9, 2022
Tailscale-client-go - A client implementation for the Tailscale HTTP API

tailscale-client-go A client implementation for the Tailscale HTTP API Example p

David Bond 9 Jun 8, 2022
Comunicación de envios de archivos entres cliente-servidor, client-client.

Client - Server - Client Estes es un proyecto simple de comunicacion de envios de archivos del cliente al servidor y viceversamente, y de cliente a cl

Melvin RB 0 Feb 11, 2022
Simple mDNS client/server library in Golang

mdns Simple mDNS client/server library in Golang. mDNS or Multicast DNS can be used to discover services on the local network without the use of an au

HashiCorp 891 Jun 29, 2022
Go Beanstalkd Client Library

#gobeanstalk Go Beanstalkd client library. INSTALL go get github.com/iwanbk/gobeanstalk USAGE Producer import ( "github.com/iwanbk/gobeanstalk" "lo

Iwan Budi Kusnanto 63 Jun 15, 2022
🌧 BitTorrent client and library in Go

rain BitTorrent client and library in Go. Running in production at put.io. Features Core protocol Fast extension Magnet links Multiple trackers UDP tr

Cenk Altı 747 Jul 2, 2022