Multitiered file storage API built on Filecoin and IPFS

Related tags

Database powergate
Overview

Powergate

Made by Textile Chat on Slack GitHub license Go Report Card GitHub action

Powergate is a multitiered file storage API built on Filecoin and IPFS, and an index builder for Filecoin data. It's designed to be modular and extensible.

Join us on our public Slack channel for news, discussions, and status updates. Check out our blog for the latest posts and announcements.

Warning This project is still pre-release and is not ready for production usage.

Table of Contents

Prerequisites

To build from source, you need to have Go 1.14 or newer installed.

Design

Powergate is composed of different modules which can be used independently, and compose together to provide other higher-level modules.

Here's a high-level overview of the main components, and how Powergate interacts with IPFS and a Filecoin client: Powergate Design

Note in the diagram that the Lotus and Filecoin client node doesn't need to be in the same host where Powergate is running. They can, but isn't necessary.

📢 Deals module

The Deals module provides a lower layer of abstraction to a Filecoin client node. It provides simple APIs to store, watch, and retrieve data in the Filecoin network. Currently, it interacts with the Lotus client but we have plans to support other Filecoin clients.

👷 Indices and Reputation scoring

Powergate builds three indexes related to on-chain and off-chain data.

The Miners index provides processed data regarding registered miners (on-chain and off-chain), such as: total miner power, relative power, online status, geolocation, and more!

The Ask index provides a fast-retrieval up to date snapshot of miner's asking prices for data storage.

The Faults index provides history data about miners faults while proving their storage on-chain.

Built on top of the previous indexes, a Reputation module constructs a weighted-scoring system that allows to sort miners considering multiple on-chain and off-chain data, such as: compared price to the median of the market, low storage-fault history, power on network, and external sources (soon!).

FFS

This module provides a multitiered file storage API built on Filecoin and IPFS. Storing data on IPFS and Filecoin is as easy as expressing your desired configuration for storing a Cid.

Want to know more about this Powergate module? Check out the FFS design document and our presentation and demo at the IPFS Pinning Summit:

Video

💫 API + CLI

Powergate exposes an API built from the various modules through gRPC endpoints. You can explore our .proto files to generate your clients, or take advange of a ready-to-use Powergate Go and JS client. 🙌

We have a CLI that supports most of Powergate features.

To build and install the CLI, run:

$ make install-pow

The binary will be placed automatically in $GOPATH/bin. You may have to set the Path variables using the below commands

$ export PATH=$PATH:$(go env GOPATH)/bin
$ export GOPATH=$(go env GOPATH)

You can then run pow in your terminal.

You can read the generated CLI docs in this repo, or run pow with the --help flag to see the available commands:

$ pow --help
A client for storage and retreival of powergate data

Usage:
  pow [flags]
  pow [command]

Available Commands:
  admin        Provides admin commands
  config       Provides commands to interact with cid storage configs
  data         Provides commands to interact with general data APIs
  deals        Provides commands to view Filecoin deal information
  help         Help about any command
  id           Returns the user id
  storage-jobs Provides commands to query for storage jobs in various states
  version      Display version information for pow and the connected server
  wallet       Provides commands about filecoin wallets

Flags:
  -h, --help                   help for pow
      --serverAddress string   address of the powergate service api (default "127.0.0.1:5002")
  -t, --token string           user auth token
  -v, --version                display version information for pow and the connected server

Use "pow [command] --help" for more information about a command.

Installation

Powergate installation involves running external dependencies, and wiring them correctly with Powergate.

External dependencies

Powergate needs external dependencies in order to provide full functionality, in particular a synced Filecoin client and a IPFS node.

Filecoin client

Currently, we support the Lotus Filecoin client but we plan to support other clients.

All described modules of Powergate need to comunicate with Lotus to build indices data, and provide storing and retrieving features in FFS. To install Lotus refer to its official documentation, taking special attention to its dependencies.

Fully syncing a Lotus node can take time, so be sure to check you're fully synced doing ./lotus sync status.

We also automatically generate a public Docker image targeting the master branch of Lotus. This image is a pristine version of Lotus, with a sidecar reverse proxy to provide external access to the containerized API. For more information, refer to textileio/lotus-build and its Dockerhub repository.

In short, a fully-synced Lotus node should be available with its API (127.0.0.1:1234, by default) port accessible to Powergate.

IPFS node

A running IPFS node is needed if you plan to use the FFS module.

If that's the case, you can refer here for installation instructions, or its Dockerhub repository if you want to run a contanerized version. Currently we're supporting v0.5.1. The API endpoint should be accessible to Powergate (port 5001, by default).

Since FFS HotStorage is pinning Cids in the IPFS node, Powergate should be the only party controlling the pinset of the node. Other systems can share the same IPFS node if can guarantee not unpinning Cids pinned by Powergate FFS instances.

Geolite database

Powergate needs an offline geo-location database to resolve miners country using their IP address. The same folder in which powd is executing, should have the Geolite2 database file GeoLite2-City.mmdb or you can pass the --maxminddbfolder flag to powd to specify the path of the folder containing GeoLite2-City.mmdb. You can copy this file from the GitHub repo at iplocation/maxmind/GeoLite2-City.mmdb. If you run Powergate using Docker, this database is bundeled in the image so isn't necessary to have extra considerations.

Server

To build and install the Powergate server, run:

make install-powd

You can run the -h flag to see the configurable flags:

$ powd -h 
Usage of powd:
      --askindexmaxparallel string       Max parallel query ask to execute while updating index (default "3")
      --askindexqueryasktimeout string   Timeout in seconds for a query ask (default "15")
      --askindexrefreshinterval string   Refresh interval measured in minutes (default "60")
      --askindexrefreshonstart           If true it will refresh the index on start
      --autocreatemasteraddr             Automatically creates & funds a master address if none is provided.
      --dealwatchpollduration string     Poll interval in seconds used by Deals Module watch to detect state changes (default "900")
      --debug                            Enable debug log level in all loggers.
      --devnet                           Indicate that will be running on an ephemeral devnet. --repopath will be autocleaned on exit.
      --disableindices                   Disable all indices updates, useful to help Lotus syncing process
      --disablenoncompliantapis          Disable APIs that may not easily comply with US law
      --ffsadmintoken string             FFS admin token for authorized APIs. If empty, the APIs will be open to the public.
      --ffsdealfinalitytimeout string    Deadline in minutes in which a deal must prove liveness changing status before considered abandoned (default "4320")
      --ffsminerselector string          Miner selector to be used by FFS: 'sr2', 'reputation' (default "sr2")
      --ffsminerselectorparams string    Miner selector configuration parameter, depends on --ffsminerselector (default "https://raw.githubusercontent.com/filecoin-project/slingshot/master/miners.json")
      --ffsminimumpiecesize string       Minimum piece size in bytes allowed to be stored in Filecoin (default "67108864")
      --ffsschedmaxparallel string       Maximum amount of Jobs executed in parallel (default "1000")
      --ffsusemasteraddr                 Use the master address as the initial address for all new FFS instances instead of creating a new unique addess for each new FFS instance.
      --gatewaybasepath string           Gateway base path. (default "/")
      --gatewayhostaddr string           Gateway host listening address. (default "0.0.0.0:7000")
      --grpchostaddr string              gRPC host listening address. (default "/ip4/0.0.0.0/tcp/5002")
      --grpcwebproxyaddr string          gRPC webproxy listening address. (default "0.0.0.0:6002")
      --ipfsapiaddr string               IPFS API endpoint multiaddress. (Optional, only needed if FFS is used) (default "/ip4/127.0.0.1/tcp/5001")
      --lotushost string                 Lotus client API endpoint multiaddress. (default "/ip4/127.0.0.1/tcp/1234")
      --lotusmasteraddr string           Existing wallet address in Lotus to be used as source of funding for new FFS instances. (Optional)
      --lotustoken string                Lotus API authorization token. This flag or --lotustoken file are mandatory.
      --lotustokenfile string            Path of a file that contains the Lotus API authorization token.
      --maxminddbfolder string           Path of the folder containing GeoLite2-City.mmdb (default ".")
      --mongodb string                   Mongo database name. (if --mongouri is used, is mandatory
      --mongouri string                  Mongo URI to connect to MongoDB database. (Optional: if empty, will use Badger)
      --repopath string                  Path of the repository where Powergate state will be saved. (default "~/.powergate")
      --walletinitialfund int            FFS initial funding transaction amount in attoFIL received by --lotusmasteraddr. (if set) (default 250000000000000000)

Localnet mode

Having a fully synced Lotus node can take a considerable amount of time and effort to mantain. We have built lotus-devnet which runs a local network with a sectorbuilder mock. This provides a fast way to spinup a local network where the sealing process if mocked, but the rest of the node logic is the same as production The localnet supports both 2Kib and 512Kib sectors, and the speed of block production is configurable. Refer to lotus-devnet readme for more information.

If you're interested in running Powergate and experiment with the CLI, the fastest way is to replace the Lotus client dependency with a running localnet, which runs a local Lotus client connected to a network with local miners.

A simple docker-compose setup is available that will run Powergate connected to a Lotus local network with 512Mib sectors and allows to use the gRPC API or CLI without any extra config flags! 🎊 Note: you will first need to install Docker compose in order to get started.

cd docker
make localnet

This will build Powergate powd, a Lotus local network with BIGSECTORS=true by default, an IPFS node and wire them correctly to be ready to use.

Note: Running BIGSECTORS=false make localnet will create the Lotus devent using 2Kib sectors. This may be more appropriate for certain development or testing scenarios.

Here is a full example of using the local network: Terminal 1:

cd docker
make localnet

Wait for seeing logs about the height of the chain increase in a regular cadence.

Terminal 2:

make build
❯ head -c 700 </dev/urandom > myfile
❯ pow admin user create
{
  "user":  {
    "id":  "c06382e0-2021-4234-be53-6e07a8d40065",
    "token":  "883f57b1-4e66-47f8-b291-7cf8b10f6370"
  }
}
❯ pow data stage -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 myfile
{
  "cid":  "QmQJxVtp61Y7UrdjUKuWvse3TxGHaPDyA7RobrBhFwqcBM"
}
❯ pow config apply -w -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm
{
  "jobId":  "b4110048-5367-4ae5-8508-709bf7969748"
}
                 JOB ID                |       STATUS       | MINER  |  PRICE   |    DEAL STATUS     
---------------------------------------+--------------------+--------+----------+--------------------
  b4110048-5367-4ae5-8508-709bf7969748 | JOB_STATUS_SUCCESS |        |          |                    
                                       |                    | f01000 | 62500000 | StorageDealActive
❯ pow data get -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm myfile2
> Success! Data written to myfile2

In this example we created a random 700 bytes file for the test, but since the localnet supports 512Mib sectors you can store store bigger files.

Production setup

A production setup is also provided in the docker folder. It launches powd connected to lotus and ipfs, plus a set of monitoring components:

  • Prometheus, which is the backend for metrics processing.
  • Grafana, for metrics dashboard.
  • cAdvisor, for container metrics.
  • Lotus, node running on the current mainnet.
  • IPFS, node running to back Powergate FFS.
  • Powergate, wired with all of above components.

Depending on which network you want to connect to, you have to run different commands:

  • make up, to connect to mainnet.

Remember that you should wait for Lotus to be fully-synced which might take a long time; you can check your current node sync status running lotus sync status inside the Lotus container. We also provide automatically generated Dockerhub images of Powergate server, see textile/powergate.

If you're interested in a more detailed explanation about Powergate installation, please refer to the installation docs.

Tests

We have a big set of tests for covering most important Powergate features.

For integration tests, we leverage our textileio/lotus-devnet configured with 2Kib sectors to provide fast iteration and CI runs.

If you want to run tests locally:

make test

It will auto-download any necessary dependencies and run all tests.

Benchmark

There's a dedicated binary to run benchmarks against a Powergate server. For more information see the specific README.

Soon we'll add benchmark results against real miners in mainnet, so stay tuned.

Contributing

This project is a work in progress. As such, there's a few things you can do right now to help out:

  • Ask questions! We'll try to help. Be sure to drop a note (on the above issue) if there is anything you'd like to work on and we'll update the issue to let others know. Also get in touch on Slack.
  • Open issues, file issues, submit pull requests!
  • Perform code reviews. More eyes will help a) speed the project along b) ensure quality and c) reduce possible future bugs.
  • Take a look at the code. Contributions here that would be most helpful are top-level comments about how it should look based on your understanding. Again, the more eyes the better.
  • Add tests. There can never be enough tests.

Before you get started, be sure to read our contributors guide and our contributor covenant code of conduct.

Changelog

Changelog is published to Releases.

License

MIT

Comments
  • Record an introduction & walk-through for Powergate v2.x

    Record an introduction & walk-through for Powergate v2.x

    We love community-produced tutorials. Here's a great example of one produced for Filecoin and Powergate:

    https://www.youtube.com/watch?v=lcgUPrGHCUY

    What do we love about this video?

    • It's thorough, but not too long. 15-20m is a great sweet-spot for a technical video like this.
    • It's technical but provides good explanations and examples to guide the viewer before getting into the methods.
    • It's clear and not rushed. Watching the video, you always have time to read or pause the video to follow along. Any code or commands have a large enough font to be visible in an embedded video.
    • It covers the right components: Filecoin, IPFS, Powergate, Localnet, and the CLI. That's the right mix to orient a new user.
    • It shows you how to build something and shares things the author learned along the way.
    • It explains the Powergate approach to storage (through configs) and how the API is used.

    Why the bounty?

    Unfortunately, this video and many others created for Powergate are now slightly out of date. We recently released Powergate v2.0 that included several updates. Even more, the video linked above was recorded before Filecoin mainnet was launched. It's time to record some new introductory content.

    Bounty expectations.

    You can use the above video as a template. You should explain the same concepts. You don't have to build the same example, but you should include a thorough tour and explanation of the CLI and a demonstration of using the API. If you create any code, your video should include a link to an open-source repo containing the content.

    Resources for your content:

    Submission Requirements

    • The video should be in high-definition.
    • The video must be posted to YouTube and Voodfy.
    • The video content should be CC licensed.
    • Code should be MIT licensed.
    • Video narration needs to be spoken and in English.
    • Submission should use the latest Powergate release at the time of creation.
    • Submission should introduce Filecoin, Powergate, Localnet, Powergate CLI, and Powergate API at a minimum. For the right "scope" for introduction to Filecoin, see the demo video linked above.
    • Video content should be accurate.

    Approval

    We will fund up to two submissions.

    For approval, we ask that you submit any information about:

    • Powergate and/or Filecoin experience.
    • Docker experience (needed for Localnet).
    • Examples of tutorials and/or videos you've created in the past.
    documentation bounty 
    opened by andrewxhill 30
  • Cannot start powergate docker

    Cannot start powergate docker

    2020-10-05T05:10:44.410Z INFO powd powd/main.go:68 starting server... 2020-10-05T05:10:44.419Z INFO server server/server.go:177 Detected Lotus node connected to network: ignition 2020-10-05T05:10:45.911Z INFO fchost fchost/fchost.go:116 connected to 6 out of 6 bootstrap peers 2020-10-05T05:10:45.912Z INFO fchost fchost/fchost.go:60 bootstraping libp2p host dht 2020-10-05T05:10:45.912Z INFO fchost fchost/fchost.go:64 dht bootstraped! 2020-10-05T05:10:45.912Z INFO server server/server.go:555 Opening badger database... 2020-10-05T05:10:46.049Z INFO server server/server.go:194 Wiring internal components... 2020-10-05T05:10:46.051Z INFO index-ask runner/runner.go:65 loaded persisted index with 283 entries 2020-10-05T05:10:46.108Z INFO index-miner module/meta.go:44 updating meta index... 2020-10-05T05:10:46.111Z INFO index-miner module/onchain.go:36 updating on-chain index... 2020-10-05T05:10:46.114Z INFO reputation reputation/reputation.go:200 rebuilding index 2020-10-05T05:10:46.114Z INFO reputation reputation/reputation.go:227 built scores for 283 miners in 0ms 2020-10-05T05:10:46.186Z FATAL powd powd/main.go:71 starting server: creating scheduler: loading scheduler trackstore: loading renewable/repairable caches: decoding cid: selected encoding not supported

    opened by kernelogic 21
  • deals: module first version

    deals: module first version

    Closes #2

    This is a first complete version of the Deals Module considering our initial implementation possibility. Some notes:

    • The Lotus client was leveraged from Lotus code. Ignore all extern folder files... they're necessary to use the Lotus client API for the moment. Plan for the future makes our own implementation (see #1), so we can wipe out the extern folder.
    • Test leverage a running local Lotus node in testnet.
    • All APIs are implemented as described in the doc.
    • Made, what I'd consider, a reasonable AvailableAsk API. Can evolve in the future.

    More comments in code.

    opened by jsign 17
  • ffs/manager: add auth token & unpublished api

    ffs/manager: add auth token & unpublished api

    Would be good to enable an admin token to call admin related APIs such as Create or other kind of APIs.

    We're living on the premise of not being exposed to the public, but that shouldn't be blocked by just adding a simple auth here. Might be wise to separate the admin endpoint to an unpublished one for extra security.

    ffs 
    opened by jsign 15
  • Docker-compose fail when try to connect to local net

    Docker-compose fail when try to connect to local net

    When I run make localnet in the powergate/docker directory I got the following error:

    powergate_1  | 2020-07-02T20:27:05.448Z	FATAL	powd	powd/main.go:63	starting server: connecting to lotus node: resolving dns: lookup lotus on 127.0.0.11:53: no such host
    localnet_powergate_1 exited with code 1
    

    I have also seen this error:

    lotus_1      | panic: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules".HandleIncomingBlocks (/go/pkg/mod/github.com/filecoin-project/[email protected]/node/modules/services.go:61): failed to build *pubsub.PubSub: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.DrandBootstrap: function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12) returned a non-nil error: lookup _dnsaddr.pl-sin.testnet.drand.sh on 127.0.0.11:53: read udp 127.0.0.1:33566->127.0.0.11:53: i/o timeout
    lotus_1      |
    lotus_1      | goroutine 1 [running]:
    lotus_1      | main.main()
    lotus_1      | 	/app/main.go:46 +0x5e0
    

    I am using a MAC

    opened by leofisG 12
  • Running server in embedded causes error

    Running server in embedded causes error

    > ./powd --embedded
    2020-04-29T00:46:24.252-0400    FATAL   powd    server/main.go:85       stat /tmp/powergate: no such file or directory
    

    Edit 1

    I manually created the directory which got me further, now it seems that it can't connect to the lotus node. Although i presume that at this point --embedded may be broken

     ./powd --embedded
    2020-04-29T00:49:54.144-0400    INFO    powd    server/main.go:112      Current configuration: 
    {
      "WalletInitialFunds": {},
      "IpfsAPIAddr": "/ip4/127.0.0.1/tcp/5001",
      "LotusAddress": "/ip4/127.0.0.1/tcp/1234",
      "LotusAuthToken": "",
      "LotusMasterAddr": "",
      "Embedded": true,
      "GrpcHostNetwork": "tcp",
      "GrpcHostAddress": "/ip4/0.0.0.0/tcp/5002",
      "GrpcServerOpts": null,
      "GrpcWebProxyAddress": "0.0.0.0:6002",
      "RepoPath": "/tmp/powergate/.powergate-083355308",
      "GatewayHostAddr": "0.0.0.0:7000"
    }
    2020-04-29T00:49:54.145-0400    INFO    powd    server/main.go:113      starting server...
    2020-04-29T00:49:54.145-0400    ERROR   powd    server/main.go:116      error starting server: connecting to lotus node: dial tcp 127.0.0.1:1234: connect: connection refused
    

    system: Ubuntu 19.10 eon

    opened by GregTheGreek 11
  • RFC: Health, Peers, Net modules

    RFC: Health, Peers, Net modules

    There are potentially a few new modules we'll want to implement. They should be generally useful and will also help support the Lotus web UI project. I want to get my thoughts about the design and implementation written down to collect feedback and come up with a final plan.

    To start, the three modules I imagine existing are Health, Peers and Net:

    • Health - Provide information about the node health. This could be derived from many sources of information.
    • Peers - Information about node peers. This will likely be augmented with derived data such as ip-based peer location. For this reason, it may be useful to think of/model this like an index similar to the existing Miners index.
    • Net - Provides general information about and methods for interacting with the network

    The first pass at these modules can easily be supported by the "Net" methods of the Lotus API:

    NetConnectedness func(context.Context, peer.ID) (network.Connectedness, error)
    NetPeers         func(context.Context) ([]peer.AddrInfo, error)
    NetConnect       func(context.Context, peer.AddrInfo) error
    NetAddrsListen   func(context.Context) (peer.AddrInfo, error)
    NetDisconnect    func(context.Context, peer.ID) error
    NetFindPeer      func(context.Context, peer.ID) (peer.AddrInfo, error)
    

    To begin, the single input to the Health module will be NetConnectedness. It's good to have Health as a module because we will add other sources of information to compute "health".

    NetPeers can be queried on some interval (chain update maybe?) and augmented with derived data to build the peers index.

    The remaining methods can be used to build the Net module with simple methods for taking the node on and offline and fining peers.

    That is my first thought on how to model this. I think it is simple but gives us some flexibility going forward to incorporate other data sources and/or adapt to other Filecoin implementations.

    opened by asutula 11
  • Fails to Make Deal with Error Not Enough Miners from Reputation Module to Satisfy The Constraints Error

    Fails to Make Deal with Error Not Enough Miners from Reputation Module to Satisfy The Constraints Error

    I am running WSL Ubuntu 20.04 (Version 2 with Docker Desktop Support) within Windows 10 Pro Version 21H1

    The steps are as follows:

    1. git clone https://github.com/textileio/powergate.git
    2. cd powergate/
    3. cd docker/
    4. nano docker-compose.yaml where I added "["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"]" between lines 32 and 33.
    5. make up
    6. Waited for the node to finish importing and then syncing.
    7. ^C then make down then deleted the line "["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"]" from docker-compose.yaml
    8. make up
    9. Now that the node was running I typed cd .. so I was in the repo's root directory, then make install-pow
    10. with the pow command in my GOPATH I typed pow to make sure pow was linked fine to powd. It was.
    11. pow admin users create
    12. copied the token and ran export POW_TOKEN=<token copied to here>
    13. Then pow wallet addrs and funded the address
    14. I went to the directory behind the folder of my static website which is about 5GB in size.
    15. I typed pow data stage <my-static-site-folder>
    16. After it was finished staging and printed out the CID I typed pow config apply --watch <CID waited a long time while it said the job was executing and then I got...
    ---------------------------------------+--------------------------------+-------+-------+--------------
                 <job id here>             | JOB_STATUS_FAILED executing    |       |       |
                                           | cold-storage config: making    |       |       |
                                           | deal configs: getting miners   |       |       |
                                           | from minerselector: getting    |       |       |
                                           | miners from reputation         |       |       |
                                           | module: not enough miners from |       |       |
                                           | reputation module to satisfy   |       |       |
                                           | the constraints                |       |       |
    

    I don't understand what the problem is. I repeated the pow config apply --watch <CID command each time adding the --override flag with several different modifications to a custom config file. The content did appear briefly on IPFS (not Filecoin), but after I continued running the config apply command the site went down from IPFS.

    Here are my powergate logs.

    <myusername>@<mycomputername>:~/dev/powergate_config_files$ docker logs mainnet_powergate_1
    2021-07-26T09:04:20.618Z        INFO    powd    powd/main.go:57 starting powd:
            version:        git
            build date:     2021-07-26T07:45:07Z
            git summary:    v2.6.2-dirty
            git branch:     master
            git commit:     34dc0fe
            git state:      dirty
    2021-07-26T09:04:20.619Z        INFO    powd    powd/main.go:72 {
      "RepoPath": "/root/powergate/.powergate",
      "MaxMindDBFolder": ".",
      "Devnet": false,
      "IpfsAPIAddr": "/dns4/ipfs/tcp/5001",
      "LotusAddress": "/dns4/lotus/tcp/1234",
      "LotusAuthToken": "<LotusAuthTokenHere>",
      "LotusMasterAddr": "",
      "LotusConnectionRetries": 180,
      "GrpcHostNetwork": "tcp",
      "GrpcHostAddress": "/ip4/0.0.0.0/tcp/5002",
      "GrpcServerOpts": null,
      "GrpcWebProxyAddress": "0.0.0.0:6002",
      "GatewayBasePath": "/",
      "GatewayHostAddr": "0.0.0.0:7000",
      "IndexRawJSONHostAddr": "0.0.0.0:8889",
      "MongoURI": "",
      "MongoDB": "",
      "FFSAdminToken": "",
      "FFSUseMasterAddr": false,
      "FFSDealFinalityTimeout": 259200000000000,
      "FFSMinimumPieceSize": 67108864,
      "FFSRetrievalNextEventTimeout": 3600000000000,
      "FFSMaxParallelDealPreparing": 2,
      "FFSGCAutomaticGCInterval": 3600000000000,
      "FFSGCStageGracePeriod": 3600000000000,
      "SchedMaxParallel": 1000,
      "MinerSelector": "reputation",
      "MinerSelectorParams": "",
      "DealWatchPollDuration": 900000000000,
      "AutocreateMasterAddr": false,
      "WalletInitialFunds": {},
      "AskIndexQueryAskTimeout": 15000000000,
      "AskindexMaxParallel": 3,
      "AskIndexRefreshInterval": 21600000000000,
      "AskIndexRefreshOnStart": false,
      "IndexMinersRefreshOnStart": false,
      "IndexMinersOnChainMaxParallel": 20,
      "IndexMinersOnChainFrequency": 21600000000000,
      "DisableIndices": false,
      "DisableNonCompliantAPIs": false
    }
    2021-07-26T09:04:20.619Z        INFO    powd    powd/main.go:75 starting server...
    2021-07-26T09:04:20.620Z        WARN    lotus-client    lotus/client.go:50      failed to connect to Lotus client dial tcp 172.20.0.3:1234: connect: connection refused, retrying...
    2021-07-26T09:04:30.621Z        WARN    lotus-client    lotus/client.go:50      failed to connect to Lotus client dial tcp 172.20.0.3:1234: connect: connection refused, retrying...
    2021-07-26T09:04:40.622Z        WARN    lotus-client    lotus/client.go:50      failed to connect to Lotus client dial tcp 172.20.0.3:1234: connect: connection refused, retrying...
    2021-07-26T09:04:50.626Z        INFO    server  server/server.go:196    Detected Lotus node connected to network: mainnet
    2021-07-26T09:05:03.233Z        INFO    fchost  fchost/fchost.go:116    connected to 11 out of 15 bootstrap peers
    2021-07-26T09:05:03.233Z        INFO    fchost  fchost/fchost.go:60     bootstraping libp2p host dht
    2021-07-26T09:05:03.233Z        INFO    fchost  fchost/fchost.go:64     dht bootstraped!
    2021-07-26T09:05:03.233Z        INFO    server  server/server.go:684    Ensuring migrations...
    2021-07-26T09:05:03.233Z        INFO    server  server/server.go:595    Opening badger database...
    2021-07-26T09:05:03.237Z        INFO    migrations      migration/migration.go:67       Current datastore version is 5, target version 5
    2021-07-26T09:05:03.237Z        INFO    server  server/server.go:699    Migrations ensured
    2021-07-26T09:05:03.239Z        INFO    server  server/server.go:595    Opening badger database...
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:217    Wiring internal components...
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:229    Starting ask index...
    2021-07-26T09:05:03.247Z        INFO    index-ask       runner/runner.go:65     loaded persisted index with 0 entries
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:235    Starting miner index...
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:247    Starting faults index...
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:256    Starting deals module...
    2021-07-26T09:05:03.247Z        INFO    deals   module/module.go:51     creating deal watcher
    2021-07-26T09:05:03.247Z        INFO    deals   module/module.go:66     resuming pending records
    2021-07-26T09:05:03.247Z        INFO    deals   module/records.go:170   resuming 0 pending records
    2021-07-26T09:05:03.247Z        INFO    deals   module/records.go:179   resumed watching 0 pending storage deal records
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:262    Starting wallet module...
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:291    Starting FFS scheduler...
    2021-07-26T09:05:03.247Z        INFO    reputation      reputation/reputation.go:200    rebuilding index
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:307    Starting gRPC, gateway and index HTTP servers...
    2021-07-26T09:05:03.247Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
    2021-07-26T09:05:03.247Z        INFO    reputation      reputation/reputation.go:227    built scores for 0 miners in 0ms
    2021-07-26T09:05:03.247Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
    2021-07-26T09:05:03.247Z        INFO    server  server/server.go:366    Starting IPFS reverse proxy...
    2021-07-26T09:05:03.248Z        INFO    deals-watcher   dealwatcher/dealwatcher.go:144  deal watcher created
    2021-07-26T09:05:03.248Z        INFO    server  server/server.go:355    Starting finished, serving requests
    2021-07-26T09:05:03.248Z        INFO    powd    powd/main.go:80 server started.
    2021-07-26T09:06:50.631Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 367
    2021-07-26T09:08:50.638Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 352
    2021-07-26T09:10:50.642Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 325
    2021-07-26T09:12:50.645Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 277
    2021-07-26T09:14:50.650Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 239
    2021-07-26T09:16:50.654Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 191
    2021-07-26T09:18:50.662Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 142
    2021-07-26T09:20:50.668Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 103
    2021-07-26T09:22:50.671Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 54
    2021-07-26T09:24:50.676Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 8
    2021-07-26T09:26:50.681Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 39, todo: 7
    2021-07-26T10:05:03.251Z        INFO    ffs-scheduler   scheduler/scheduler.go:208      running scheduler gc...
    2021-07-26T10:05:03.259Z        INFO    ffs-scheduler   scheduler/scheduler.go:236      scheduler gc ran with 0 excluded cids, unpinning 0 staged cids
    2021-07-26T10:35:18.496Z        INFO    ffs-manager     manager/manager.go:109  creating instance
    2021-07-26T10:35:18.528Z        INFO    ffs-auth        auth/auth.go:38 generating auth-token for instance <instanceIdHere>
    2021-07-26T11:00:21.977Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Pushing new configuration...
    2021-07-26T11:00:21.995Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Configuration saved successfully
    2021-07-26T11:00:21.995Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 1, total executing: 0
    2021-07-26T11:00:22.005Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 1
    2021-07-26T11:00:22.006Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Executing job <JobIdHere>...
    2021-07-26T11:00:22.009Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Automatically staging Cid from the IPFS network...
    2021-07-26T11:00:22.019Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Executing Cold-Storage configuration...
    2021-07-26T11:00:22.022Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Current replication factor is lower than desired, making 1 new deals...
    2021-07-26T11:00:22.024Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Entering deal preprocessing queue...
    2021-07-26T11:00:22.026Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Calculating piece size...
    2021-07-26T11:01:36.270Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       The payload size is 5.1 GiB, and the calculated piece size is 8.0 GiB
    2021-07-26T11:01:36.278Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Starting deals failed, with cause: making deal configs: getting miners from minerselector: getting miners from reputation module: not enough miners from reputation module to satisfy the constraints
    2021-07-26T11:01:36.280Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Cold-Storage execution failed.
    2021-07-26T11:01:36.282Z        ERROR   ffs-scheduler   scheduler/scheduler.go:531      executing job <JobIdHere>: executing cold-storage config: making deal configs: getting miners from minerselector: getting miners from reputation module: not enough miners from reputation module to satisfy the constraints
    2021-07-26T11:01:36.285Z        INFO    ffs-sched-sjstore       sjstore/sjstore.go:702  notifying watcher
    2021-07-26T11:01:36.285Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Job <JobIdHere> execution failed: executing cold-storage config: making deal configs: getting miners from minerselector: getting miners from reputation module: not enough miners from reputation module to satisfy the constraints
    2021-07-26T11:01:36.287Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
    2021-07-26T11:01:36.288Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
    2021-07-26T11:01:36.288Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
    <myusername>@<mycomputername>:~/dev/powergate_config_files$
    

    Here are my IPFS logs:

    <myusername>@<mycomputername>:~/dev/powergate_config_files$ docker logs mainnet_ipfs_1
    Changing user to ipfs
    ipfs version 0.7.0
    Found IPFS fs-repo at /data/ipfs
    Initializing daemon...
    go-ipfs version: 0.7.0-ea77213
    Repo version: 10
    System version: amd64/linux
    Golang version: go1.14.4
    Swarm listening on /ip4/127.0.0.1/tcp/4001
    Swarm listening on /ip4/127.0.0.1/udp/4001/quic
    Swarm listening on /ip4/172.20.0.2/tcp/4001
    Swarm listening on /ip4/172.20.0.2/udp/4001/quic
    Swarm listening on /p2p-circuit
    Swarm announcing /ip4/127.0.0.1/tcp/4001
    Swarm announcing /ip4/127.0.0.1/udp/4001/quic
    Swarm announcing /ip4/172.20.0.2/tcp/4001
    Swarm announcing /ip4/172.20.0.2/udp/4001/quic
    Swarm announcing /ip4/68.80.241.181/udp/55230/quic
    API server listening on /ip4/0.0.0.0/tcp/5001
    WebUI: http://0.0.0.0:5001/webui
    Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
    Daemon is ready
    <myusername>@<mycomputername>:~/dev/powergate_config_files$
    

    Please let me know what the problem is, I really love Filecoin and Powergate and want to take part in the ecosystem.

    opened by ChristianOConnor 10
  • Client & CLI updates for Health & Net

    Client & CLI updates for Health & Net

    Pretty straight forward. Also added tests for Health & Net client code. I want to make sure the pattern I'm using for testing is good, then we can expand the client tests next.

    opened by asutula 9
  • Unable to build powergate CLI

    Unable to build powergate CLI

    Followed the steps mentioned in https://docs.textile.io/powergate/ to clone and build Powergate CLI. I am facing following errors and unable to proceed.

    My environment is M1 macbook Pro with golang version go version go1.18 darwin/amd64

    On running make install-pow following error is displayed.

    go: downloading github.com/marten-seemann/qtls-go1-17 v0.1.0-rc.1
    go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
    # github.com/lucas-clemente/quic-go/internal/qtls
    ../../../go/pkg/mod/github.com/lucas-clemente/[email protected]/internal/qtls/go118.go:5:13: cannot use "quic-go doesn't build on Go 1.18 yet." (untyped string constant) as int value in variable declaration
    make: *** [install-pow] Error 2
    

    Thought i found a workaround mentioned here https://github.com/ipfs/go-ipfs/issues/8819 . Tried to proceed with it but I am running into the below error. Looks like go-ipfs version used is 0.8 which is very old and using some flags which have been removed in later dependency (quic-go).

    CGO_ENABLED=0 go install -ldflags="-X github.com/textileio/powergate/v2/buildinfo.GitSummary=v2.6.2-6-gb82b011-dirty -X github.com/textileio/powergate/v2/buildinfo.Version=git -X github.com/textileio/powergate/v2/buildinfo.BuildDate=2022-05-12T00:08:11Z -X github.com/textileio/powergate/v2/buildinfo.GitCommit=b82b011 -X github.com/textileio/powergate/v2/buildinfo.GitBranch=master -X github.com/textileio/powergate/v2/buildinfo.GitState=dirty" ./cmd/pow
    # github.com/ipfs/go-ipfs/core/node/libp2p
    ../../../go/pkg/mod/github.com/ipfs/[email protected]/core/node/libp2p/transport.go:21:7: undefined: quic.RetireBugBackwardsCompatibilityMode
    make: *** [install-pow] Error 2
    
    opened by chaitanyaprem 8
  • Add build caching to docker

    Add build caching to docker

    Still trying to understand how exactly this works and what exactly BuildKit is, but this simple change seems too good to be true. Caches build objects and speeds up docker builds by many many times. Based on what I was reading in this blog post and the others in the same series: https://www.docker.com/blog/containerize-your-go-developer-environment-part-2/

    opened by asutula 8
  • Bump google.golang.org/grpc from 1.36.1 to 1.49.0

    Bump google.golang.org/grpc from 1.36.1 to 1.49.0

    Bumps google.golang.org/grpc from 1.36.1 to 1.49.0.

    Release notes

    Sourced from google.golang.org/grpc's releases.

    Release 1.49.0

    New Features

    • gcp/observability: add support for Environment Variable GRPC_CONFIG_OBSERVABILITY_JSON (#5525)
    • gcp/observability: add support for custom tags (#5565)

    Behavior Changes

    • server: reduce log level from Warning to Info for early connection establishment errors (#5524)

    Bug Fixes

    • client: fix race in flow control that could lead to unexpected EOF errors (#5494)
    • client: fix a race that could cause RPCs to time out instead of failing more quickly with UNAVAILABLE (#5503)
    • client & server: fix a panic caused by passing a nil stats handler to grpc.WithStatsHandler or grpc.StatsHandler (#5543)
    • transport/server: fix a race that could cause a stray header to be sent (#5513)
    • balancer: give precedence to IDLE over TRANSIENT_FAILURE when aggregating connectivity state (#5473)
    • xds/xdsclient: request correct resource name when user specifies a new style resource name with empty authority (#5488)
    • xds/xdsclient: NACK endpoint resources with zero weight (#5560)
    • xds/xdsclient: fix bug that would reset resource version information after ADS stream restart (#5422)
    • xds/xdsclient: fix goroutine leaks when load reporting is enabled (#5505)
    • xds/ringhash: fix config update processing to recreate ring and picker when min/max ring size changes (#5557)
    • xds/ringhash: avoid recreating subChannels when update doesn't change address weight information (#5431)
    • xds/priority: fix bug which could cause priority LB to block all traffic after a config update (#5549)
    • xds: fix bug when environment variable GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION is set to true (#5537)

    Release 1.48.0

    Bug Fixes

    • xds/priority: fix bug that could prevent higher priorities from receiving config updates (#5417)
    • RLS load balancer: don't propagate the status code returned on control plane RPCs to data plane RPCs (#5400)

    New Features

    • stats: add support for multiple stats handlers in a single client or server (#5347)
    • gcp/observability: add experimental OpenCensus tracing/metrics support (#5372)
    • xds: enable aggregate and logical DNS clusters by default (#5380)
    • credentials/google (for xds): support xdstp C2P cluster names (#5399)

    Release 1.47.0

    New Features

    • xds: add support for RBAC metadata invert matchers (#5345)

    Bug Fixes

    • client: fix a context leaked if a connection to an address is lost before it is fully established (#5337)
    • client: fix potential panic during RPC retries (#5323)

    ... (truncated)

    Commits
    • 1c29e07 Change version to 1.49.0 (#5583)
    • 8e5a84e xds/resolver: generate channel ID randomly (#5603)
    • 92cee34 gcp/observability: Add logging filters for logging, tracing, and metrics API ...
    • c7fe135 O11Y: Added support for custom tags (#5565)
    • 7981af4 test/kokoro: add missing image tagging to the xDS interop url map buildscript...
    • 6f34b7a xdsclient: NACK endpoint resource if load_balancing_weight is specified and i...
    • f9409d3 ringhash: handle config updates properly (#5557)
    • 946dde0 xdsclient: NACK endpoint resources with zero weight (#5560)
    • b89f49b xdsclient: deflake Test/LDSWatch_PartialValid (#5552)
    • 9bc72de grpc: remove mentions of WithBalancerName from comments (#5555)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies minor 
    opened by dependabot[bot] 0
  • Bump github.com/ipfs/go-ipfs from 0.8.0 to 0.13.1

    Bump github.com/ipfs/go-ipfs from 0.8.0 to 0.13.1

    Bumps github.com/ipfs/go-ipfs from 0.8.0 to 0.13.1.

    Release notes

    Sourced from github.com/ipfs/go-ipfs's releases.

    v0.13.1

    go-ipfs v0.13.1 Release

    This release includes security fixes for various DOS vectors when importing untrusted user input with ipfs dag import and the v0/dag/import endpoint.

    View the linked security advisory for more information.

    Changelog

    • github.com/ipfs/go-ipfs:
      • chore: update car
    • github.com/ipld/go-car (v0.3.2 -> v0.4.0) & (v2.1.1 -> v2.4.0):
      • Bump version in prep for releasing go-car v0
      • Revert changes to insertionindex
      • Revert changes to index.Index while keeping most of security fixes
      • Return error when section length is invalid varint
      • Drop repeated package name from CarStats
      • Benchmark Reader.Inspect with and without hash validation
      • Use consistent CID mismatch error in Inspect and BlockReader.Next
      • Use streaming APIs to verify the hash of blocks in CAR Inspect
      • test: add fuzzing for reader#Inspect
      • feat: add block hash validation to Inspect()
      • feat: add Reader#Inspect() function to check basic validity of a CAR and return stats
      • Remove support for ForEach enumeration from car-index-sorted
      • Use a fix code as the multihash code for CarIndexSorted
      • Fix testutil assertion logic and update index generation tests
      • fix: tighter constraint of singleWidthIndex width, add index recommentation docs
      • fix: explicitly disable serialization of insertionindex
      • feat: MaxAllowed{Header,Section}Size option
      • feat: MaxAllowedSectionSize default to 32M
      • fix: use CidFromReader() which has overread and OOM protection
      • fix: staticcheck catches
      • fix: revert to internalio.NewOffsetReadSeeker in Reader#IndexReader
      • fix index comparisons
      • feat: Refactor indexes to put storage considerations on consumers
      • test: v2 add fuzzing of the index
      • fix: v2 don't divide by zero in width indexes
      • fix: v2 don't allocate indexes too big
      • test: v2 add fuzzing to Reader
      • fix: v2 don't accept overflowing offsets while reading v2 headers
      • test: v2 add fuzzing to BlockReader
      • fix: v2 don't OOM if the header size is too big
      • test: add fuzzing of NewCarReader
      • fix: do bound check while checking for CIDv0
      • fix: don't OOM if the header size is too big
      • Add API to regenerate index from CARv1 or CARv2

    ... (truncated)

    Changelog

    Sourced from github.com/ipfs/go-ipfs's changelog.

    v0.13.1 2022-07-06

    This release includes security fixes for various DOS vectors when importing untrusted user input with ipfs dag import and the v0/dag/import endpoint.

    View the linked security advisory for more information.

    Changelog

    • github.com/ipfs/go-ipfs:
      • chore: update car
    • github.com/ipld/go-car (v0.3.2 -> v0.4.0) & (v2.1.1 -> v2.4.0):
      • Bump version in prep for releasing go-car v0
      • Revert changes to insertionindex
      • Revert changes to index.Index while keeping most of security fixes
      • Return error when section length is invalid varint
      • Drop repeated package name from CarStats
      • Benchmark Reader.Inspect with and without hash validation
      • Use consistent CID mismatch error in Inspect and BlockReader.Next
      • Use streaming APIs to verify the hash of blocks in CAR Inspect
      • test: add fuzzing for reader#Inspect
      • feat: add block hash validation to Inspect()
      • feat: add Reader#Inspect() function to check basic validity of a CAR and return stats
      • Remove support for ForEach enumeration from car-index-sorted
      • Use a fix code as the multihash code for CarIndexSorted
      • Fix testutil assertion logic and update index generation tests
      • fix: tighter constraint of singleWidthIndex width, add index recommentation docs
      • fix: explicitly disable serialization of insertionindex
      • feat: MaxAllowed{Header,Section}Size option
      • feat: MaxAllowedSectionSize default to 32M
      • fix: use CidFromReader() which has overread and OOM protection
      • fix: staticcheck catches
      • fix: revert to internalio.NewOffsetReadSeeker in Reader#IndexReader
      • fix index comparisons
      • feat: Refactor indexes to put storage considerations on consumers
      • test: v2 add fuzzing of the index
      • fix: v2 don't divide by zero in width indexes
      • fix: v2 don't allocate indexes too big
      • test: v2 add fuzzing to Reader
      • fix: v2 don't accept overflowing offsets while reading v2 headers
      • test: v2 add fuzzing to BlockReader
      • fix: v2 don't OOM if the header size is too big
      • test: add fuzzing of NewCarReader
      • fix: do bound check while checking for CIDv0
      • fix: don't OOM if the header size is too big
      • Add API to regenerate index from CARv1 or CARv2
      • PrototypeChooser support (#305) (ipld/go-car#305)
      • bump to newer blockstore err not found (#301) (ipld/go-car#301)
      • Car command supports for largebytes nodes (#296) (ipld/go-car#296)
      • fix(test): rootless fixture should have no roots, not null roots

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.
    dependencies minor 
    opened by dependabot[bot] 0
  • Bump github.com/muesli/termenv from 0.7.4 to 0.12.0

    Bump github.com/muesli/termenv from 0.7.4 to 0.12.0

    Bumps github.com/muesli/termenv from 0.7.4 to 0.12.0.

    Release notes

    Sourced from github.com/muesli/termenv's releases.

    v0.12.0

    What's Changed

    New Contributors

    Full Changelog: https://github.com/muesli/termenv/compare/v0.11.0...v0.12.0

    v0.11.0

    What's Changed

    Full Changelog: https://github.com/muesli/termenv/compare/v0.10.0...v0.11.0

    v0.10.0

    What's Changed

    New Contributors

    Full Changelog: https://github.com/muesli/termenv/compare/v0.9.0...v0.10.0

    v0.9.0

    Changes:

    • Use HSLuv color space for color profile conversions

    Added:

    • Add explicit constants and functions for clearing lines

    v0.8.1

    ... (truncated)

    Commits
    • 0899c83 fix: shims for AIX and Plan9
    • 321ab8e feat: update x/sys
    • 44cd139 Support foreground detection on Solaris/illumos
    • 662a4e4 Add documentation about EnvColorProfile
    • ce3deab screen sessions should always support ANSI256
    • 6964eb8 Fix DisableMousePress doing the opposite
    • 5ac8409 Don't use OSC queries if we're not in control of the terminal
    • afdc360 Add Windows cmd to terminal support list
    • 4e0b03c Add Windows color support detection
    • 3129ed3 Reorder README paragraphs
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies minor 
    opened by dependabot[bot] 0
  • Bump github.com/ory/dockertest/v3 from 3.6.3 to 3.9.1

    Bump github.com/ory/dockertest/v3 from 3.6.3 to 3.9.1

    Bumps github.com/ory/dockertest/v3 from 3.6.3 to 3.9.1.

    Commits
    • 7b25128 chore(deps): bump github.com/opencontainers/runc from 1.1.1 to 1.1.2 (#355)
    • 148c3da chore(deps): bump gotest.tools/v3 from 3.1.0 to 3.2.0 (#353)
    • e38b974 chore: containerd/continuity (#352)
    • cfab898 chore(deps): bump github.com/opencontainers/runc from 1.1.0 to 1.1.1 (#350)
    • f6f31ef chore(deps): bump github.com/cenkalti/backoff/v4 from 4.1.2 to 4.1.3 (#351)
    • 230039c fix: resource host for rootless containers (#348)
    • 23d5405 chore(deps): bump github.com/docker/cli (#349)
    • d93907d chore(deps): bump github.com/docker/cli (#345)
    • ef54b32 feat: add platform field to run options to support Apple M1 (#344)
    • 593d31d chore(deps): bump github.com/Microsoft/go-winio from 0.5.1 to 0.5.2 (#340)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies minor 
    opened by dependabot[bot] 0
  • Bump github.com/filecoin-project/go-fil-markets from 1.8.1 to 1.20.3

    Bump github.com/filecoin-project/go-fil-markets from 1.8.1 to 1.20.3

    Bumps github.com/filecoin-project/go-fil-markets from 1.8.1 to 1.20.3.

    Changelog

    Sourced from github.com/filecoin-project/go-fil-markets's changelog.

    go-fil-markets v1.20.3

    Contributors

    Contributor Commits Lines ± Files Changed
    Anton Evangelatov 1 +8/-0 1

    go-fil-markets v1.20.1

    Contributors

    Contributor Commits Lines ± Files Changed
    Masih H. Derkani 1 +34/-28 4

    go-fil-markets v1.20.0

    Contributors

    Contributor Commits Lines ± Files Changed
    Aarsh Shah 1 +890/-219 22

    go-fil-markets v1.19.1

    Contributors

    Contributor Commits Lines ± Files Changed
    Hannah Howard 1 +23/-5 6

    go-fil-markets v1.19.0

    Contributors

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies minor 
    opened by dependabot[bot] 0
Releases(v2.6.2)
Owner
textile.io
IPFS and Filecoin tools.
textile.io
A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Fei Chen 851 Sep 11, 2022
Simple-read-file - Example of how to read file in Go

simple-read-file This repository contains a simple example of how to read file i

Yessy Purnama 0 Jan 11, 2022
A GPU-powered real-time analytics storage and query engine.

AresDB AresDB is a GPU-powered real-time analytics storage and query engine. It features low query latency, high data freshness and highly efficient i

Uber Open Source 2.9k Sep 9, 2022
IceFireDB - Distributed disk storage system based on Raft and RESP protocol.

Distributed disk storage database based on Raft and Redis protocol.

GITSRC 921 Sep 24, 2022
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library. moss stands for "memory-oriented s

null 886 Sep 15, 2022
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice. A simple in-memory database and table implementation are provided, and you can query any data source you want by implementing a few interfaces.

DoltHub 891 Sep 22, 2022
Key-Value Storage written in Go.

kvs kvs is an in-memory key-value storage written in Go. It has 2 different usage. It can be used as a package by importing it to your code or as a se

Gökhan Özeloğlu 4 Jun 15, 2022
The lightweight, distributed relational database built on SQLite.

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it grace

rqlite 11k Sep 18, 2022
Golang in-memory database built on immutable radix trees

go-memdb Provides the memdb package that implements a simple in-memory database built on immutable radix trees. The database provides Atomicity, Consi

HashiCorp 2.5k Sep 17, 2022
Launch parallel processes (shuttles) with conditional parameter(s) specified in text file(s)

~# shuttle Launch some shuttles here and there. Usage Usage example: # Launch as many sqlmaps as the lines in targets.txt # with 4 maximum istances at

5amu 0 Jan 8, 2022
A rest-api that works with golang as an in-memory key value store

Rest API Service in GOLANG A rest-api that works with golang as an in-memory key value store Usage Run command below in terminal in project directory.

sercan aydın 0 Dec 6, 2021
A simple golang api generator that stores struct fields in key/value based databases

Backgen A simple golang API generator that uses key/value based databases. It does not provide the database itself, only uses a interface to access se

null 0 Feb 4, 2022
Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

Welcome to NIPO Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on command

Morteza Bashsiz 16 Jun 13, 2022
Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures.

Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures. capabilities which owl provides include Process approval、sql Audit、sql execute and execute as crontab、data backup and recover .

null 35 Jun 17, 2022
Being played at The Coffee House and try to find and play it on Spotify

The Coffee House Muzik Follow the music that is being played at The Coffee House and try to find and play it on Spotify. Installation Clone this proje

SangND 4 May 25, 2022
Walrus - Fast, Secure and Reliable System Backup, Set up in Minutes.

Walrus is a fast, secure and reliable backup system suitable for modern infrastructure. With walrus, you can backup services like MySQL, PostgreSQL, Redis, etcd or a complete directory with a short interval and low overhead. It supports AWS S3, digitalocean spaces and any S3-compatible object storage service.

Ahmed 446 Sep 18, 2022
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

James Mills 9 Aug 24, 2022
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Josh Baker 3.9k Sep 21, 2022
Concurrency-safe Go caching library with expiration capabilities and access counters

cache2go Concurrency-safe golang caching library with expiration capabilities. Installation Make sure you have a working Go environment (Go 1.2 or hig

Christian Muehlhaeuser 1.8k Sep 23, 2022