Powerful Blockchain streaming data engine, based on StreamingFast Firehose technology.

Overview

Substreams - A streaming data engine for The Graph - by StreamingFast

DEVELOPER PREVIEW OF SUBSTREAMS

Think Fluvio for deterministic blockchain data.

The successor of https://github.com/streamingfast/sparkle, enabling greater composability, yet similar powers of parallelisation, and a much simpler model to work with.

Install client

This client will allow you to interact with Substreams endpoints, and stream data in real-time.

Get a release.

From source:

git clone [email protected]:streamingfast/substreams
cd substreams
go install -v ./cmd/substreams

From source without checkout:

go install github.com/streamingfast/substreams/cmd/[email protected]

Install dependencies to build Substreams

This will allow you to develop Substreams modules locally, and run them remotely.

Install rust

We're going to be using the Rust programming language, to develop some custom logic.

There are several ways to install Rust, but for the sake of brevity:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Run remotely

Using StreamingFast's infrastructure

Dump that somewhere like .bashrc:

export STREAMINGFAST_KEY=server_YOUR_KEY_HERE  # Ask us on Discord for a key
function sftoken {
    export FIREHOSE_API_TOKEN=$(curl https://auth.dfuse.io/v1/auth/issue -s --data-binary '{"api_key":"'$STREAMINGFAST_KEY'"}' | jq -r .token)
	export SUBSTREAMS_API_TOKEN=$FIREHOSE_API_TOKEN
    echo Token set on FIREHOSE_API_TOKEN and SUBSTREAMS_API_TOKEN
}

Then in your shell, load a key in an env var with:

sftoken

Then, try to run the PancakeSwap Substreams from our Substreams Playground project

The below commands will be run from substreams-playground

cd ./pcs-rust/ && ./build.sh
cd ../eth-token/ && ./build.sh
cd ..
substreams run -e bsc-dev.streamingfast.io:443 ./pcs-rust/substreams.yaml pairs,block_to_pairs,volumes,totals,db_out -s 6810706 -t 6810711

Run locally

You can run the substreams service locally this way:

Get a recent release of the Ethereum Firehose, and install sfeth.

Alternatively, you can use this Docker image: ghcr.io/streamingfast/sf-ethereum:6aa70ca, known to work with version v0.0.5-beta of the substreams release herein.

Get some data (merged blocks) to play with locally (here on BSC mainnet):

# Downloads 2.6GB of data
sfeth tools download-from-firehose bsc-dev.streamingfast.io:443 6810000 6820000 ./localblocks
sfeth tools generate-irreversible-index ./localblocks ./localirr 6810000 6819700

Then run the firehose service locally in a terminal, reading blocks from your disk:

sfeth start firehose  --config-file=  --log-to-file=false  --common-blockstream-addr=  --common-blocks-store-url=./localdata --firehose-grpc-listen-addr=:9000* --substreams-enabled --substreams-rpc-endpoint=https://URL.POINTING.TO.A.BSC.ARCHIVE.NODE/if-you/want-to-use/eth_call/within/substreams

And then run the substreams command against your local deployment (checkout substreams-playground in the Run remotely section above):

substreams run -k -e localhost:9000 wasm_substreams_manifest.yaml pairs,block_to_pairs,db_out,volumes,totals -s 6810706 -t 6810711
Issues
  • publish protos to bsr

    publish protos to bsr

    Let's publish all protos on the buf.build BSR.

    We might also want to use the buf.gen.yaml to use the remote builders to generate the prost and go code.

    The external protos from google could be declared as deps.

    Just putting this here because I had to (well, for convenience) publish these to BSR for generate code for the typescript / node.js consumer example

    opened by fubhy 1
  • fix run remotely

    fix run remotely

    otherwise

    $ substreams run -e bsc-dev.streamingfast.io:443 ./pcs-rust/substreams.yaml pairs,block_to_pairs,volumes,totals,db_out -s 6810706 -t 6810711
    2022-05-10T09:49:48.613+0800 (substreams) starting atomic level switcher {"listen_addr": "127.0.0.1:1065"}
    Error: read manifest "./pcs-rust/substreams.yaml": reading file "./pkg/pcs_substreams_bg.wasm": open ./pkg/pcs_substreams_bg.wasm: no such file or directory
    
    opened by 0xbe1 1
  • Implement state builder setter interfaces

    Implement state builder setter interfaces

    	SetMaxBigInt(ord uint64, key string, value *big.Int)
    }
    type MaxInt64Setter interface {
    	SetMaxInt64(ord uint64, key string, value int64)
    }
    type MaxFloat64Setter interface {
    	SetMaxFloat64(ord uint64, key string, value float64)
    }
    type MaxBigFloatSetter interface {
    	SetMaxBigFloat(ord uint64, key string, value *big.Float)
    }
    
    type MinBigIntSetter interface {
    	SetMinBigInt(ord uint64, key string, value *big.Int)
    }
    type MinInt64Setter interface {
    	SetMinInt64(ord uint64, key string, value int64)
    }
    type MinFloat64Setter interface {
    	SetMinFloat64(ord uint64, key string, value float64)
    }
    type MinBigFloatSetter interface {
    	SetMinBigFloat(ord uint64, key string, value *big.Float)
    }
    
    type SumBigIntSetter interface {
    	SumBigInt(ord uint64, key string, value *big.Int)
    }
    type SumInt64Setter interface {
    	SumInt64(ord uint64, key string, value int64)
    }
    type SumFloat64Setter interface {
    	SumFloat64(ord uint64, key string, value float64)
    }
    type SumBigFloatSetter interface {
    	SumBigFloat(ord uint64, key string, value *big.Float)
    }```
    enhancement 
    opened by colindickson 1
  • StreamingFast: ReadME

    StreamingFast: ReadME

    Remove the second paragraph of "Features", touting Search and Lifecycle as products. Remove all "Top-tier products" in the Overview section. Update "Protocols", to include Solana, NEAR. Start with Ethereum. Also include as "Contributed by third-parties", the ones by Figment (https://github.com/figment-networks/firehose-cosmos which includes the Cosmos Hub and Osmosis network). Remove the EOSIO one from the Protocols list. Links to Docs should point to: the FIrehose docs (firehose.streamingfast.io) and Substreams (to substreams.streamingfast.io) Remove the "Common interfaces" section. Grab Google's CODE_OF_CONDUCT doc from somewhere, and tweak it to reflect our name, and put it in this repository. When we have a code of conduct linked somewhere, we can ask GitBook for a free "open-source project" license. Change the "about" and "website" section of the GitHub repository, to point to streamingfast.io and our latest company name.

    documentation 
    opened by jubeless 0
  • Substreams: Ethereum doc review, revamp

    Substreams: Ethereum doc review, revamp

    At https://docs.rs/substreams-ethereum/latest/substreams_ethereum/ the first page doesn't demonstrate what that package can do When digging in abigen (but why would you go there?) https://docs.rs/substreams-ethereum/latest/substreams_ethereum/struct.Abigen.html you find some things you can copy&paste and use. Check the documentation conventions of Rust, and start doing it that way. I know for instance, that packages need a top-level doc, like what we have for https://docs.rs/substreams .. That top-level doc for https://docs.rs/substreams needs to be sync'd with the latest ways of doing things. I'm not sure the examples therein would compile, or are not missing some imports that you'll be looking for if you copy&pasted those examples. There's docs in the sub-package https://docs.rs/substreams/latest/substreams/handlers/attr.store.html and https://docs.rs/substreams/latest/substreams/handlers/attr.map.html how much of those are duplicated with the front page? Which section is easier to maintain (and check that it compiles, etc..) Perhaps something simpler on the front page, and details in each map/store packages? And more direct links to those

    documentation 
    opened by jubeless 0
  • Substreams: Rust API

    Substreams: Rust API

    Review the Rust APIs section, and its interaction with the documentation living in the docs.rs website. https://substreams.streamingfast.io/reference-and-specs/rust-api

    Perhaps the links should point to the docs directly, instead of crates.io, (or in a table view with a direct link to the docs). The links should be without version, so they are directed to the latest version: https://docs.rs/substreams-ethereum and https://docs.rs/substreams ...

    documentation 
    opened by jubeless 0
  • Substreams: README

    Substreams: README

    Review the top-level README.md therein, and see what's still useful, and if the links are not broken. Beware this repository is the one under GitBook management.

    documentation 
    opened by jubeless 0
  • Sustreams: Change History Review

    Sustreams: Change History Review

    Review the change history of the Substreams repository https://github.com/streamingfast/substreams and ask the team what's new, find a way to present the new things in the change-log, sync the GitHub releases page with what's new.

    Let's plan together how we could best communicate what is new. Our release process is not so strictly defined, so we can figure out what's best together.

    documentation 
    opened by jubeless 0
Releases(v0.0.13)
  • v0.0.13(Jun 3, 2022)

    CLI

    • Changed substreams manifest info -> substreams info
    • Changed substreams manifest graph -> substreams graph
    • Updated usage

    Changelog

    • 43e9bd7 Added a test with modInit starting at 0.
    • b89c9c8 Added comment to --start-block. Fix ranges splitter tests.
    • e5d3aaf Don't early kill the scheduler if it didn't fail.
    • db8780d Don't send BlockScopedData when we're doing backprocessing.
    • 13084a4 Fix tests.
    • b0dfd11 Fixed issue with ranges.
    • cc6c420 Major rework of the scheduling plan, called SplitWork, called WorkUnit, tobe renamed.
    • cdce6e5 Merge remote-tracking branch 'origin/develop' into splitter
    • fb9c955 Revise usage and CLI docs.
    • 21e5772 Simplify the scheduler exit loop.
    • e346177 SplitWork implementation, responsible for driving the show in Squasher and Strategy.
    • 152de88 Squasher / Waiter / Queue / RequestPool / Strategy Simplified. Hanging bug fixed, which was caused by the Squasher never preparing the store when no work was required.
    • 0a00c03 Unlock, but what does that test do anyway?
    • b42926d Updated change-log.
    • 8c4805f Updated some element of the docs
    • e203658 Updated substreams info
    • 8d82cb2 Whoops, don't print "" on 'enter'.. misfeature :)
    • 2520a0b substreams info and substreams graph now.
    • f752f3c block range: split occurs on chunk size boundary
    • 069d028 orchestrator test: make adding to queue concurrent
    • c75a8ba queue: add more testing
    • 054021e splitter wip
    • e761111 squasher: fix test
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(311 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.13_linux_x86_64.tar.gz(8.15 MB)
    substreams_0.0.13_macOS_arm64.tar.gz(8.31 MB)
    substreams_0.0.13_macOS_x86_64.tar.gz(8.54 MB)
  • v0.0.12(Jun 3, 2022)

    Summary

    • Fix null pointer exception at the end of CLI run in some cases.

    • Do log last error when the CLI exit with an error has the error is already printed to the user and it creates a weird behavior.

    Changelog

    • b10966f Improved error reporting in CLI
    • 7b30088 Preparing for release 0.0.12
    • 13b2945 Updated releaase notes for v0.0.11
    • bcebc40 remove cancel() if prog is nil
    • 65b3740 returning ctx.Err() if ctx cancelled caught in the scheduler. caused issue that an error would be missed to be caught at top level
    • 721aeb1 worker pool: streamline contexts
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(311 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.12_linux_x86_64.tar.gz(8.15 MB)
    substreams_0.0.12_macOS_arm64.tar.gz(8.31 MB)
    substreams_0.0.12_macOS_x86_64.tar.gz(8.54 MB)
  • v0.0.11(Jun 3, 2022)

    Changelog

    • 27872dc Added mean on top level README (all linking to direct docs links) and reduce emphasis on Developer Preview
    • e631024 Added updates rate in CLI.
    • 105768c Better boundaries for job scheduling.
    • 20bd9e4 Better failure capture when a module fails.
    • c67dbe1 Exclude GitBook: commits from showing up in the changelog
    • d615bad Fixed all doctests, remove all ignores, move docs in substreams-macro directly to substreams::handlers
    • 936023c Merge remote-tracking branch 'origin/develop' into develop
    • 32d4183 Permit arguments to be passed to docker image on run directly
    • 916fc14 Preparing for 0.0.11 release
    • 43f0c59 Prioritize first ranges
    • 682c864 Send the writing of the output cache into a goroutine
    • b98ca77 better log
    • 363fa03 crate: substreams bumped version
    • 95dec45 fix squashing partial range
    • 1204cc1 fix squashing partial range again
    • dc65d53 pad block nums in tui
    • d16f0ae release terminal on ctrl+c
    • bd9d4d6 state: add retry for writing state to dstore, normalize derr retry to 3 everywhere
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(311 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.11_linux_x86_64.tar.gz(8.15 MB)
    substreams_0.0.11_macOS_arm64.tar.gz(8.31 MB)
    substreams_0.0.11_macOS_x86_64.tar.gz(8.54 MB)
  • v0.0.10-beta(Jun 2, 2022)

    Changelog

    • 25a5b5d "state: keep delete of merge data for compatibility of files"
    • 20bd9e4 Better failure capture when a module fails.
    • 86e1db1 Fix log propagation between instances
    • 009a6b7 GitBook: [#62] No subject
    • c9dd9a2 GitBook: [#63] No subject
    • b4266f1 GitBook: [#64] No subject
    • d583d50 GitBook: [#65] No subject
    • 0b2036c GitBook: [#67] No subject
    • fd56dd5 GitBook: [#68] No subject
    • 8fab00b GitBook: [#69] more-additions
    • aa3827c GitBook: [#70] No subject
    • 7d84c92 GitBook: [#71] No subject
    • 5f13b4f GitBook: [#73] No subject
    • cf516ca GitBook: [#74] No subject
    • 5e293b1 Ok, now solved the issue. By simplifying!
    • 10fc034 Reset the instances between blocks.
    • a5c6797 small fix for subsequent executions of the same substreams which caused edge case issue and not detecting the stores where ready
    • 74c7410 state: output value types all normalized to lowercase
    • 8277aa4 state: remove merge data special key. no longer used
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(326 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.10-beta_linux_x86_64.tar.gz(8.15 MB)
    substreams_0.0.10-beta_macOS_arm64.tar.gz(8.31 MB)
    substreams_0.0.10-beta_macOS_x86_64.tar.gz(8.54 MB)
  • v0.0.9-beta(Jun 2, 2022)

    Changelog

    substreams CLI

    • Added substreams run -o ui|json|jsonl|module-json|module-jsonl.

    Commits

    • 5a4a193 Add a clause to write a temporary KV when not on boundaries.
    • eb53701 Added a bit of time between crate publishing
    • 5cd7e55 Added hex_literal macro re-export
    • 6fc452a Added more documentation
    • c282412 Align bounds properly.
    • 66c27c6 Always clean-up terminal
    • 3887379 An attempt to fix hanging, improved
    • 893b9b4 Bar graph! Hit 'm' to see :)
    • 1ba6af7 Better TUI
    • eb83089 Better clean-up and work on progress.
    • 322aa2a Better mermaid graph.
    • 6b38234 Better range calculations.
    • 79d46e7 Better ranges again.
    • ee0665e Better var name
    • 40ff1a0 COmment
    • a517c8d Cosmetics
    • 2a287f3 Default to BarMode. A few refactors.
    • 14e5358 Do error checking on incoming manifests, client and server.
    • 64616b3 First draft at bubbletea integration. Takes control of keyboard mon enfouaré.
    • 296f8e3 Fix change log
    • 60e2148 Fix documentation setup
    • 6c09bf2 Fix more range management issues.
    • fc1f855 Fixed small typo in docs
    • 79cf059 Fixup latest tests.
    • 8f32e9f Fixup progress.
    • c12c7ef Fixup some race between channels.
    • 2d8ba92 Fixup the squashing world
    • ff0b8a8 Fixup which ranges we're using
    • 6c99b8e Found a subtle bug that impeched the merging :)
    • d98928e Four new output modes
    • 92ef8fc Improve ranges again
    • a9cd3ff Incredible progress :)
    • 8569066 Merge commit 'ee0665ef71b94a1d10f72805fffd42f9b0c61956' into develop
    • 1f0962d Merge remote-tracking branch 'origin/develop' into develop
    • 37970e3 Properly replace the stores after a backprocessing shot.
    • 5d263fe Quick rename.
    • c95e01e Remove README by default
    • 1e228ce Remove the BlockRange on the Store
    • 731dc47 Removed extraneous import
    • 0b8f5e5 Removed metadata section in README.md
    • dff1fdd Removed progress package, now lives in the TUI. Repacked decode and print into a more formal and expansive TUI.
    • 55332d8 Removed unused code
    • 05d34b4 Rename
    • bce17d6 Revert "small fix on kv getting reinitialized on each new squasher for each store"
    • fd2c258 Simplify output
    • 8434b97 Small update to docs
    • eb6e21b SynchronizeStore to output the stores when they are built.
    • a631908 Updated to use tagged release of dependencies in docs
    • b8ba02c When we know there aren't requests from the STRATEGY we early exit.
    • ad68e06 Working version for SF Ethereum: * Need to compute the merge ranges better.
    • a1d7a32 Works!!! Oh well :)
    • 109259b adding tests for merge block ranges
    • d019739 doc: testing layout and file structure
    • 2a545dd fix back process not returning when no back process request need to be run
    • 8d8ef94 fixing waiter tests
    • a61c1ff getting out of the go func when we get and error from the worker of a scheduler
    • ff0041a orchestrator: add stringer methods for debugging, waiter does not wait for stores if its initial block is gt desired block num
    • d6f776e orchestrator: add tests
    • f1f7352 orchestrator: concurrnt squasher test
    • fcd2a88 orchestrator: fix squash test
    • e0df979 orchestrator: refactoring waiter condition, update strategy test
    • 4f5da66 pool: add log for notify
    • 3a1a8ca renaming stuff and adding logs
    • 96bf4e9 replacing some println statements for some zlog and fixing partial/kv boundary condition
    • 17c685d small fix for run cmd
    • 941dfe5 small fix on kv getting reinitialized on each new squasher for each store
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(323 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.9-beta_linux_x86_64.tar.gz(8.15 MB)
    substreams_0.0.9-beta_macOS_arm64.tar.gz(8.31 MB)
    substreams_0.0.9-beta_macOS_x86_64.tar.gz(8.54 MB)
  • v0.0.8-beta(May 30, 2022)

    Changelog

    • a9384e3 Adjusted release script to be more robust and correclty run publish commands in dry mode
    • 409f4c1 Prepare for 0.0.8-beta release
    • 6a986f2 Replaced rust/substreams/build.rs with explicit generation using protoc-gen-prost plugin
    • 4f68088 Updated to for next release, removed statik again
    • 8159f8d fix builder kv not initialize
    • dc9a3ca now failed when a store is reference as input with a wrong name
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(323 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.8-beta_linux_x86_64.tar.gz(8.09 MB)
    substreams_0.0.8-beta_macOS_arm64.tar.gz(8.24 MB)
    substreams_0.0.8-beta_macOS_x86_64.tar.gz(8.46 MB)
  • v0.0.7-beta(May 30, 2022)

    Summary

    • Changed startBlock to initialBlock in substreams.yaml manifests.

    • code: is now defined in the binaries section of the manifest, instead of in each module. A module can select which binary with the binary: field on the Module definition.

    • Added substreams inspect ./substreams.yaml or inspect some.spkg to see what's inside. Requires protoc to be installed (which you should have anyway).

    • Added command substreams protogen that writes a temporary buf.gen.yaml and generates Rust structs based on the contents of the provided manifest or package.

    • Added substreams::handlers macros to reduce boilerplate when create substream modules.

      substreams::handlers::map is used for the handlers corresponding to modules of type map. Modules of type map should return a Result where the error is of type Error

      /// Map module example
      #[substreams::handlers::map]
      pub fn map_module_func(blk: eth::Block) -> Result<erc721::Transfers, Error> {
           ...
      }
      

      substreams::handlers::store is used for the handlers corresponding to modules of type store. Modules of type store should have no return value.

      /// Map module example
      #[substreams::handlers::store]
      pub fn store_module(transfers: erc721::Transfers, s: store::StoreAddInt64, pairs: store::StoreGet, tokens: store::StoreGet) {
            ...
      }
      

    Changelog

    • eceec63 Added --exclude-paths flag on substream protocgen
    • 4945217 Added Substreams crate documentation
    • 82ae6b7 Added protogen
    • c02a5b4 Added some missing well known proto descriptors in the system.pb, removed statik replaced by
    • 621ba88 Added support for the sf.susbtreams.v1.Clock in inputs.
    • 1a779ca Added top-level doc.. really thin though :)
    • 42517ea Avoid panic in state builder when missing deltas is encountered
    • 81a642c Better print
    • c9742a0 Bunch of renames and refactors. BREAK BACKWARDS COMPATIBILITY. Needs to be redeployed everywhere. Packages need to be rebuilt and repackaged. New substreams needs to be used to interact with server.
    • db26ea7 Change all those store methods helpers to be u64 instead of i64, as that's the type we use in all ordinals all over the place (Eth, and Solana).
    • 153b2cf Cleaned up some unecessary deps
    • ce857aa Cleaned up substream lib and exported macros
    • c7debad Correctly handle error on call to grpcClientFactory
    • d380853 Doc fixes.
    • 29bdd3f Ensure that tests passes in rust code
    • f9a32d9 First draft at documentaiton.
    • 9dca46b Fix manifest info formatting
    • 280b9e1 Fix pack output message
    • 5545177 Fix proto import dependancy
    • 134a5f7 Fix typo
    • ec40b6c Fixed up docs. Added set_many for a few.
    • 59215b0 Fixed wrong version in Cargo on substreams crate
    • c2633a4 Fixup protogen
    • c260fca Fixup range.
    • 0286a7f Fixup some retries.
    • 0dc3c0a GitBook: [#10] No subject
    • 68efa88 GitBook: [#12] No subject
    • f4ab67a GitBook: [#13] Modules and packages
    • cd2ae90 GitBook: [#20] No subject
    • 43158ee GitBook: [#23] getting-started
    • ee01b07 GitBook: [#24] Better docs, merge strategies, improvements to modules.
    • bf2ee07 GitBook: [#25] getting-started
    • 709ee88 GitBook: [#26] No subject
    • 07382c8 GitBook: [#27] No subject
    • 26cddf4 GitBook: [#28] getting-started
    • 931c9de GitBook: [#29] No subject
    • dae3ffc GitBook: [#2] title modification
    • 97e2c49 GitBook: [#4] Substreams doc revamp
    • ef4cf64 GitBook: [#5] No subject
    • 408ed30 Improved hex encoder and added decoding from hex library
    • 9aec010 Merge branch 'develop' into wasm-refactor
    • 59f299e Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 4ddec88 Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • c78ffc2 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
    • eac4b1b Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
    • d9d39d9 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
    • e9da6a8 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
    • b5730c9 Merge branch 'feature/parallel_partial_sub_req' of github.com:streamingfast/substreams into feature/parallel_partial_sub_req
    • 5766c32 Merge branch 'feature/publish' into develop
    • deabc1f Merge branch 'manifest-reader' into develop
    • e7412d3 Merge branch 'rust-macros' into develop
    • 77fe753 Merge branch 'wasm-refactor' into develop
    • c174526 Merge pull request #13 from streamingfast/feature/parallel_partial_sub_req
    • 3682925 Merge pull request #14 from streamingfast/feature/broken_cache
    • af7288b One response func instead of many.
    • f023ca0 Preparing for release of CLI + crate
    • 654b58e Refactor substream macros
    • c885e3e Refactored instrensiv names
    • c3fcf05 Remove --partial-mode and --no-return-handler .. we can put --partial-mode back if necessary to ease debugging of backend services. We could use SUBSTREAMS_PARTIAL_MODE=true, as to not bloat the end user's experience.
    • 89c8a0f Remove ETH reference
    • cd4a4f6 Remove a Grrr
    • 67dafd8 Remove the Ethereum stuff
    • f3f52dc Removed blank lines in errors
    • dd2dd59 Renamed loadDetla to loadDeltas, replaced some panic with errors, updated some zlog statements
    • 450855f Review the Progress signaling.
    • c0e477d Some renames
    • 4ed5ae8 Take out Partial from the proto.
    • 45c5647 Take out non working install instructions.
    • 836ef82 Try again
    • 1b2672c Try to pad things. Fix the proto import priority.
    • ff67ce5 Trying to fix cargo publish command
    • 70d042e Tweaked docs.
    • cd1bd78 Updated CHANGELOG
    • 33abcc2 Updated CHANGELOG, with substream macros
    • 06f8f35 Updated README.md
    • 9309b53 Updated Summary
    • eb8171a Updated explicit traits
    • b74178a Updated latest rendered substreams version
    • ca3c4b0 Updated packages metadata
    • 3883630 WIP
    • beb843e WIP: cleaned up args
    • 8ffb631 Working
    • 3b3e008 Wrap up protogen. Fix tests.
    • 76b46f4 added an early exit when no catchup request are needed
    • 893e016 added couple of MarshalLogObject func
    • 5b8cf2d added lock to squasher
    • 1cecae2 added logs to debug context stuff
    • 381202c added missing return
    • 8edee92 added missing return worker to pool
    • 1da7057 added retry around dstore access
    • 6c92140 added retry on sub request
    • 3c4fd6a added some log
    • 2261fa9 added unlock to squasher ;-)
    • 022b2b7 adding config for scheduler's subrequests and block range size
    • 228b828 adding information on the progress bar
    • fe447ba adding log for request created and adding nil check on if the partial file doesn't exist
    • 7b47ce6 adding missing pointer for squasher in scheduler struct
    • 47e11a7 adding pipeline and scheduler wip
    • e3ff44c adding progress print for catch up of parallel requests
    • 54b8c54 adding return for scheduler workers
    • 70e1c39 better logging
    • 6345d61 better logging and grpc stream handling
    • a1e1517 better logging and remove sched config
    • 9c1e030 broken wip
    • 9629eab builder, etc: replace builder start block and end block with block range object
    • ed57b8a change block range split algo
    • 77001af clean up save stores snapshots
    • e654b3c cleaned up fix macros
    • 3eb410e client: fix package name overwrite
    • 339fed9 delete partial file after squash
    • 2767c43 do not reuse grpc client create a new for each sub request
    • 30b4096 fix "substreams-partial-mode" request header better logging fix early exit of workers
    • 1387ff2 fix NO_RETURN_HANDLER
    • 996c163 fix all merge tests
    • d5768da fix delta output
    • 1076c00 fix error message for max subrequest range size
    • cdbbb22 fix init builder block range
    • 1b5bf42 fix init builder block range
    • 09883e1 fix initialize store so only module outputs one can be initialized has partial fix builder initialization with deltas
    • 62790e9 fix load deltas when cache file is empty
    • 24869bb fix load state info
    • 420c037 fix logs
    • df92975 fix output caching
    • b76b0de fix partial file naming
    • 327d094 fix range split
    • 2d4c509 fix reader
    • c01e1bd fix request pop order from list
    • 78daa28 fix retry code scope
    • cd64eeb fix scheduler error handling
    • 4682ce6 fix some retry code
    • 83ecdc6 fix squasher not finding the right file
    • 3393f9b fix squashing
    • 41ee189 fix summary layout
    • 8270603 fix test
    • 54f2fa5 fix test
    • 89f8a2a fix test and remove Squash func from builder
    • e8898d5 fix worker threading issue
    • 63fa967 fixing mermaid
    • b28b287 flip builder merger func
    • a8778d5 handle grpc client getter error
    • a11432c merge develop
    • d8adcb3 merge values into struct
    • 587ce1b more logs
    • 783b45e move scheduler and squasher to orchestrator package
    • 082aa25 move some things around
    • 5f49840 moving stuff around and creating own concept for progress, instead of having 2 different files doing similarly the same thing
    • d6865d7 new global worker pool
    • 8252c5b orchestrator refactoring
    • fd9c3de orchestrator: add basic first test for squasher
    • 8fdc673 orchestrator: add priority queue to pool to re-order requests in the wait buffer
    • 057389c orchestrator: better context handling
    • 1915fa7 orchestrator: change lock to rw mutex
    • 6ebca76 orchestrator: first draft of ordered strategy
    • 6793cdc orchestrator: fix pool tests
    • bd2c674 orchestrator: fix race conditions in pool
    • 8c55852 orchestrator: fixed close of closed channel on waiter item, switch pipeline to ordered strategy
    • f237c49 orchestrator: more testing
    • 7abd562 orchestrator: more tests for block waiter
    • 28abc3a orchestrator: pass map of already processed blocks to Waiter for early exit condition
    • 791efcd orchestrator: priority queue now uses csp instead of polling, with some additional renamings and cleanup
    • 193ba09 orchestrator: renames
    • e65e8fa orchestrator: request stream handles context errors
    • d2b4d1a orchestrator: smarter locking for priority queue
    • 5888a3a orchestrator: tests and refactorings
    • a0b473d padding computation
    • adc78f4 pipeline: InitializeStores uses different store methods depending on partial mode
    • efafe7e pipeline: add option to limit size of subrequest range
    • 7c62709 pipeline: call squasher.Close
    • 5fa453f pool: add comment
    • 50e0c6a pool: use rw-lock in notify method
    • 420db82 prevent context override
    • f1aff5d proto: add partial mode to request, pipeline does not syncronize stores in partial mode
    • 5e127ea range: code cleanup
    • 63d16b6 refactor scheduler: use strategy
    • ec0cd7e refactor to adapt to new worker pool changes
    • 093cc59 remove build()unused parameters
    • 80292b6 remove build()unused parameters 2
    • 5116a00 remove bytes option when generating proto
    • 1a412a7 remove dum log and comments
    • e0386a2 remove todo
    • 5c86a8f removing bar thingy
    • 1d6b58b removing useless var
    • 5b3a354 scheduler squasher wip fun
    • ec0f358 service: remove old unused worker pool
    • 301482b squash is no more deleting partial files
    • 41f1c67 squashable: fix bug checking first item of empty ranges slice
    • 33e1c8f squasher do not failed anymore when not able to delete partial file
    • 1de7727 squasher first edition
    • f6de7e6 squasher first release candidate
    • 9163124 squasher: code structure for clarity
    • 84f1b47 squasher: refactor to add some clarity
    • 7774ab7 state: fix info save
    • a0e90a3 state: split up massive builder.go file into more manageable pieces
    • c1cdb4e state: update MarshalLogObject
    • 1a480f0 upgrading progress and not including first range for the squasher
    • 6fa7396 use ordered list of state builders instead of map
    • a6983ef wip
    • ef1dbe9 wip
    • 615f6dc wip
    • 9e5e9e5 wip
    • 3b2ec56 wip
    • 160678b wip
    • 45b0e37 wip
    • 2380fb5 wip
    • e76394e wip for squasher changes
    • 33155db worker pool borrow refactoring
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(323 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.7-beta_linux_x86_64.tar.gz(8.09 MB)
    substreams_0.0.7-beta_macOS_arm64.tar.gz(8.24 MB)
    substreams_0.0.7-beta_macOS_x86_64.tar.gz(8.46 MB)
  • v0.0.6-beta(May 17, 2022)

    Changelog

    • b4df491 Add strings to beginning of signature.
    • 5112914 Added Semver check for package version.
    • d986c5d Added --stop-block count support
    • f3b0293 Added log::debug! and log::info! and improved log::println
    • 20a3646 Added substreams::Hex wrapper type to more easily deal with printing and encoding bytes to hexadecimal string
    • 8797355 Added reading of SF_API_TOKEN also if notjing else finding
    • 627ab02 Added validation for names. Merge for proto, packages, rename modules and implement package imports.
    • c4d2d41 Fix delta start block
    • 8cb54cf Fixed Docker image and improved its building speed
    • a799cfd Fixed Dockerfile
    • 2502a0b Fixed Slack notification
    • fb17a73 Fixup required/optional flags.
    • 86af019 Handle error, and better error reporting for packing.
    • 16812f9 Improved around logs handling and reporting
    • beb3616 Improved image around rust installation
    • 552ed05 Improved readability of substreams run
    • b6f3aaf Merge branch 'feature/new_store_sync' into develop
    • 1ef72a8 Merge branch 'feature/packaging' into develop
    • b8c3b6e Package vs Manifest vs Modules.
    • 9169b1b Remove RPC for Eth, not the concern of this repo. Make import paths absolute.
    • 3954a1b Remove left over debug log
    • b51ed28 Removed wasm-pack from substreams
    • ea114bd Removed wasm-pack mentions
    • 2b7b28c Support http and https for package dependencies.
    • b120ea8 The -c (or --compact-output) can be used to print JSON as a single compact line
    • 3148278 The whole thing builds now!
    • 87030db Trap context canceled error when WASMExtension are executed
    • 568e2d4 Try to dedupe some protobuf files based on Sum256 of its serialized payload. Ideally, we'd use the latest, so some reverse sort of some kind, so newer stuff takes precedence on older stuff.
    • c3273b7 Updated Dockerfile to have a better image
    • b848c7a Works! .spkg acts as a substreams.yaml :)
    • 4f3a55e add 'tools package' commands: get,list,info,create
    • 43e788d add protobuf descriptor.proto to statik fileDescriptorSet
    • 55ec114 added a grrr log
    • 621dac0 eee wut, fix typos
    • f9650b0 fix caching issues
    • 39008a5 fix init builder block range
    • ad8d17d fix output cache by encoding it into json
    • 3413675 fix start-block missing in command line to pick up from the manifest and streaming till live if no stop-block is set
    • 9e78226 fix temp removed code
    • cd29802 pipeline: blocksFunc split into chunks of 10k block ranges
    • 7d80854 remove 'cli', put all in cmd
    • 91b8549 remove mode name from all stored file remove baseCacheStore params remove all hardcode 10_000 value
    • bb43944 remove my packaging tools
    • 0b30f78 removing mermaid print on run of substreams
    • 4ce57ac replacing wasmer-go from wasmerio with streamingfast wasmer-go
    • e2f31d5 service: worker pool moved to struct-level
    • 0804fe6 start block validation
    • 6372493 sub requested for store synchronization load state at any block from kv and deltas data folder re-org
    • 2a83fe5 tools: fix store checker
    • 449cec2 wip for bundling proto files
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(323 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.6-beta_linux_x86_64.tar.gz(8.11 MB)
    substreams_0.0.6-beta_macOS_arm64.tar.gz(8.26 MB)
    substreams_0.0.6-beta_macOS_x86_64.tar.gz(8.48 MB)
  • v0.0.5-beta3(May 9, 2022)

    Changelog

    • 0d517aa Added Docker build, GitHub Actions and improved Go released with brew
    • 61cc596 Fixed wrong detection of errors
    • 7862c07 Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 0f09be3 Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • b2cbf5e Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 60d2d07 Merge pull request #11 from streamingfast/no-more-store
    • 0920aff Updated CHANGELOG
    • 190f31c added last todo for store deltas output caching
    • 95f1d59 added store delta to module output caching
    • 767b111 adding String method to Store and Mapper executor, adding optimize executors and adding tests for optimize executors
    • ff2f5b0 applying deltas on store deltas for caching
    • 5434ac1 bringing back native code
    • ecc7a84 bringing back oneof native code
    • 9d2239a cleaup tests
    • 5e53cc8 convert many if statement to switch
    • 2160ab7 extracted code to func
    • 95e3d4e fix all tests for @colin
    • f1cc0ad fix new cache handling
    • 5b7ff61 fix save builder block range handling
    • a1e517a move code around
    • a862cae moving stuff around
    • ef193e6 pipeline: added ctx to sync blocks function
    • 687dd98 pipeline: remove possibility of dupilicate requests in synchronize blocks
    • 2984cf8 pipeline: synchronize blocks
    • 79bb8c4 refactor executor
    • 67e3d8d remove executor optimization
    • a63c91e remove wait from pipeline
    • 7d0c4f9 small cache registration optimization
    • 00c425b split executor and pipeline
    • 4775db9 split store on build if it is a delta or not
    • 95cb811 state: refactor deleting of special keys
    • 6d3edf3 super wip
    • b7f5cff tools: fix bug in checker
    • 17b5448 tools: remove merge cmd and added check cmd
    • 1637eed wip
    • 3935a0f wip for output cache
    • 8a5e739 working output caches
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(326 bytes)
    checksums.txt.sig(134 bytes)
    substreams_0.0.5-beta3_linux_x86_64.tar.gz(8.07 MB)
    substreams_0.0.5-beta3_macOS_arm64.tar.gz(8.21 MB)
    substreams_0.0.5-beta3_macOS_x86_64.tar.gz(8.45 MB)
  • v0.0.5-beta2(May 4, 2022)

    Changelog

    Client

    • Improved defaults for --proto-path and --proto, using globs.
    • WASM file paths in substreams.yaml manifests now resolve relative to the location of the yaml file.
    • Added substreams manifest package to create .pb packages to simplify querying using other languages. See the python example.
    • Added substreams manifest graph to show the Mermaid graph alone.
    • Improved mermaid graph layout.
    • Removed native Go code support for now.

    Server

    • Always writes store snapshots, each 10,000 blocks.
    • A few tools to manage partial snapshots under substreams tools

    Commits

    • 05012e1 Added substreams manifest package command.
    • 0770804 Added good defaults for proto rendering experience.
    • 5adb936 Always add .pb and that's it.
    • e9677eb Improve mermaid graph. Added substreams manifest graph
    • cadb0e4 Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 26175e8 Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 5712290 Merge branch 'develop' of github.com:streamingfast/substreams into pipeline-rework
    • de77bd4 No need to install wasmer
    • d829192 Remove eth check, open up possibility for Solana.
    • b09403f Rework of pipeline, module executor, in prep for caching
    • 3222703 Update README.md
    • aa8436d Updated .gitignore
    • 35e1f4a Updated README.md
    • fb25f4d Updated README.md
    • 36eed17 added module name to state store folders fix some tests
    • dd1ff1e adding wip module ouput cache
    • 911b189 better error message when substreams terminate with error remove println
    • 215f243 cmd: added first version of squasher
    • 09b5bd4 database changes have been moved, adding validation for input sources
    • a606476 find proto files recursively and change manifest module code path to absolute
    • df77c1f fix panic handling
    • b5d922e lower level of some logs
    • 1652f15 make pipeline compile and test to runn
    • 03d1b6c more log to debug
    • 0463c3a only accepted value for input.source is sf.ethereum.type.v1.Block, in other cases, we return an error
    • 77d5015 pipeline: allow invalid state in partial mode
    • 99f3cda pipeline: fix snapshot interval
    • ca3f41d remove database changes stuff from substreams
    • 8ff8f25 remove database from proto build
    • 6f767ff remove database proto from substreams
    • 4041ba7 remove println again
    • 5b63a0a remove un-used code
    • 484cd9c removing a comment
    • 91e2048 removing test
    • c95415a renaming of parameter
    • a31eab5 setting remote cmd to root as we only have remote command available
    • 8923491 spec out module output cache
    • 49ee4fc squasher cmd, state.ParseFile refactor
    • c2625d0 squasher: fix args
    • 0a083d6 state: fix bug where BuilderFromFile would create an additional substore
    • 2bc44ae store sync todos
    • d715952 substreams run commend
    • a0ec093 todo management
    • 74bae47 tools: cleanup in parallel
    • 4c7931a uncommented panic recovery code in pineline
    • 6d4fe90 update readme
    • 067cb55 update readme
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(219 bytes)
    substreams_0.0.5-beta2_Darwin_arm64.tar.gz(8.23 MB)
    substreams_0.0.5-beta2_Darwin_x86_64.tar.gz(8.47 MB)
  • v0.0.5-beta(Apr 27, 2022)

    [v0.0.5-beta]

    First chain-agnostic release. THIS IS BETA SOFTWARE. USE AT YOUR OWN RISK. WE PROVIDE NO BACKWARDS COMPATIBILITY GUARANTEES FOR THIS RELEASE.

    See https://github.com/streamingfast/substreams for usage docs.

    • Removed local command. See README.md for instructions on how to run locally now. Build sfeth from source for now.
    • Changed the remote command to run.
    • Changed run command's --substreams-api-key-envvar flag to ``--substreams-api-token-envvar, and its default value is changed fromSUBSTREAMS_API_KEYtoSUBSTREAMS_API_TOKEN`. See README.md to learn how to obtain such tokens.

    Previous Changelog

    • 1a22a92 Added CHANGELOG.md
    • 378d759 Added LICENSE
    • 6f6e46a Added README.md
    • c6e537a Added notes.
    • b2e180d Added some docs. Renamed remote to run.
    • 84b6852 Adding Cargo lock to gitignore
    • 961d855 Adding decimals
    • 86b760d Adding proto wip
    • 408277e Adding sum float 64
    • 60b5f59 Adding tests and adding sum_float64
    • 6b9e5ab Apply table changes to an entity
    • 54b7a68 Attempt at integrating protobuf to json rendering upon output.
    • bb77f30 Better flags setting.
    • 2b128c7 Better manifest
    • b47ad60 Better print
    • 91e1cd4 Bump prost to 0.10.1
    • b246896 Bumped tidwall/gjson to fix sec issue.
    • 39ac0f2 Change hard code value of block
    • 856060c Changelog update. First draft of goreleaser.
    • c396a0f Changing variable names and adding logging
    • 2b2abce Closer closer to something functional. Missing the decoding and displaying the rest is pretty much there.
    • ae1b219 Compiles. No more deps on sparkle now.
    • 05e7960 Decode faster, support unsupported messages.
    • 79583a7 Do the decoding at the right place, and load the proto-path.
    • 1fbdfac Don't allow running local when not built through sf-ethereum. Only remote.
    • 3f31be3 Don't conditionally add the getters
    • f29022a Don't fail when proto not available. Show connecting/connected.
    • f19fde7 Draft of substreams-native gRPC endpoint.
    • bfc1c94 First draft at importing from PCS and splitting in a proper lib
    • e753428 First draft at moving proto defs around, to have a first-class Substreams endpoint.
    • a792613 First draft of proto for manifests
    • cb97e74 First workable draft of multi-module remote sync'ing.
    • 655782f Fix imports
    • ac1c28d Fix leak between blocks for input sources.
    • d47606c Fix tests. SetBuidlder -> SetOutputStore. We need the data to align for bigint
    • 853deea Fixed graph building, and renamed a few stream things into modules.
    • 1610439 Fixup RPC calls
    • 310e60a Fixup printing of those blocks.
    • 38b71c8 Fixup rust tests.
    • 4526a22 Fixup the registry, and finding the right func.
    • c3b725e Fixups
    • 2b242c9 Flush stores at stop block too. Fix compilation issues. Fix logs. Options here and there. Cosmetics in builder.go
    • 2f33f70 Go generated code
    • 4fd59bb Go mod tidy
    • 7c144c3 Gracefully terminate. Better print output. Optimize anypb.Any creation.
    • 8f73094 Implementing and adding tests for set max int64, bigint, flaot64 and bigfloat
    • 0a5fb9d Implementing sum_big_float and adding tests
    • c18fa29 Improved docs. Renamed remote to run.
    • fc6b067 Initialize the wasm modules within the Firehose thread.
    • 03cef88 LocalConfig and RemoteConfig now extends Config
    • 785889e Mega hack!
    • 19ac176 Merge
    • 64a2135 Merge branch 'develop' into feature/proto_manifest
    • 025a55f Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • aa8db1e Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • fd2ebbb Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 947015b Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 835a1cc Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 5cc72f2 Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • fb55150 Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 0165628 Merge branch 'develop' of github.com:streamingfast/substreams into develop
    • 09b405c Merge branch 'develop' of github.com:streamingfast/substreams into feature/refactor-state-imports
    • 11efad5 Merge pull request #1 from streamingfast/master
    • 4925463 Merge pull request #6 from streamingfast/feature/waiter_refactoring
    • 67b9fab Merge pull request #7 from streamingfast/refactor-instance
    • 44eac00 Merge pull request #8 from streamingfast/graphnode_returnhandler
    • 31a8cb4 ModuleSignature WIP
    • 8fa5ae4 Move return handler out of there. Will be able to be re-used by the front-end to do the same sort of decoding there.
    • 5c747ea Much better on-the-fly decoding
    • a377ef7 Par dessus
    • 19d3238 Pass in deltas (serialized as JSON for now)
    • 48aecaa Proper fallback on ENV variables.
    • 81bd08b REVERT! now use block payload instead of unmarshal and re marshall it
    • 2951140 Recovered cache baby, somehow, baby tukka.
    • 3f00f60 Removing time handling log
    • 2a47bb9 Rename a few proto stuff
    • f8586cc Rename all those freaking fields. Fix mermaid output. Make that state deltas proto buf serialized instead of json.
    • a037fe2 Renames
    • 6a9136f Restore println's output. Cleaner output.
    • 43b99b4 Revert "Initialize the wasm modules within the Firehose thread."
    • db70091 Revert "remove all panic"
    • c6d7524 Revert "test commit"
    • ba27538 Safeguard against WASM data leaking between blocks
    • b36d168 Shared engine
    • 521a8e8 Sign native code too.
    • d85c2f5 Small refactor. Implement SetIfNotExists. Write as strings in storage for ints and floats. Runtime error when stores do not match imported function calls.
    • a96619f Start block int64. Better print.
    • 10cbc89 Try to pinpoint the odd behavior
    • 5287ec4 UPdate
    • 4340f80 Update README.md
    • 7674fe3 Update the rust package.
    • 5964ae4 Updated README.md
    • 0528605 Updated loggers for graph-node and rpc packages
    • ce9bbdc Updated proto
    • 3c37fbb Use protoreflect for dynamic proto message handling.
    • 0a7a75c Use startBlock from the manifest now.
    • 013ab56 Using require isEqual with testing suite
    • fe9c283 WIP
    • 312c326 WIP exchange dataloader
    • e40a039 WIP for bigfloat
    • 63a8106 WIP transform, print
    • 104e7ea WIP: have decode/print.go print with the new output format.
    • 4f77f84 Wire in the cursor, step still missing for the remote transforms-based version.
    • 54bff23 add a more verbose format and resetting time to now in if statement
    • c7fb4cd add context to progress tracker
    • db8b90b add default case to input mode and set keys to lowercase
    • 11e0c44 add eth_call to rpc to wasm machine, using proto
    • 54be505 add gen for substreams
    • 41d63da add log for rpc call time taken
    • fa6ba49 add logger module
    • 570c145 add memory leak test
    • 48076b9 add missing file
    • 0554f76 add prost-build in build dependencies
    • 2a21aff add proto generate files and build scripts
    • a4eb426 add protobuf
    • 2b4fb7e add remote runtime, update cli, cleanup runtime vars
    • a4a0405 add set_if_not_exist as extern function
    • e42dc23 add tests for graph methods from manifest file and fix some tests
    • 348cc22 add time blocks pps
    • b5f493e add time to process x number of blocks per second
    • df7c321 add timer for rpc
    • 9f00432 added a small tool to print some manifest info
    • 7077a9b added graph-node definition parsing
    • af02435 added initial block to module and start block is now a program argument
    • bd02d03 added no-return-handler param
    • ecf7264 added param rpc-cache-store-url
    • 50ac81c added secondary-rpc-endpoints params
    • d2a95ef added set_min support!
    • 9fe1dd1 added stack traces log when recovery from panic
    • d08b4a8 added stupid fmt.println
    • 7f411f8 added substreams Output "wrapper" around return value
    • d25dd6b added support to multi rpc endpoints
    • 6e5887b added test for graph
    • b53083a added unset to table_change operation
    • e5608b7 adding logging for types v1 package
    • 5ca3e6a adding more logs and adding statesSaveInterval
    • f1cb290 adding proto url for local command
    • f6f5e93 adding protobufBlockType missing in the LocalConfig constructor
    • 97fbd4b adding test orphan node in the test file
    • f295730 all broken wip
    • 7791c31 better logging
    • c6f764c better start block handling
    • af01a46 builder io: walk partials func
    • 6bf377d builder: compile-time check for interfaces
    • 05bbd05 builder: fix call to WriteState
    • 40555a6 builder: fix tests
    • 1154df0 builder: fix tests
    • 92d9dba builder: if partial file starts at module start block, save as full snapshot
    • 25521a5 builder: implement merge logic for all types (int64, bigInt, float64, big.Float)
    • 982de94 builder: max tests
    • ef1eb93 builder: min tests
    • 352a891 builder: partial prep work wip
    • b60b87b builder: remove bundler stuff
    • 96e64bd builder: remove patate log
    • df30de1 builder: sum tests
    • 2a44a23 builder: unify receiver names
    • 84f4eac bump bstream, add ground work for substreams as transform
    • a448af6 bump bstream, firehose with new transform patterns
    • c77f822 bump dstore
    • 515e830 bump dstore
    • ae83b68 bump eth-go
    • 2889dd3 bump eth-go, now uses context
    • c972832 bump substreams to pick up bugfix for gas uint64 deterministic error
    • 8db7cc9 change module initialStartBlock to startBlock
    • 9a3f3fe check requested start block before wait loop
    • dcfa9c9 cherry pick changes from substreams_transform branch and implement onReturn method
    • cbc2063 clear todo
    • 94c1c35 cli: add command to get ordered list of stores to run in parallel
    • a5f31be cli: add stop-block flag
    • af5e1f8 create a new store for each instance
    • 315403d database changes test update
    • 0d3fd72 database squash test cleanup
    • e1e8f43 debug/info -> println
    • de8c53c deps: use streamingfast fork of graph library
    • 5cde373 ensure substreams transform runner catches EOF as non-error
    • ce5cb6e ensure transform creates an (empty) cache to prevent nil pointer
    • 9013de9 entity package copied over from sparkle
    • 3132c4a extern state methods are no longer taking owner ship of given value
    • 914371c extra state import resgister to a func
    • 5a74e7c fix ModulesDownTo
    • 5e17d6d fix SynchronizeStores
    • 8afe606 fix argument count
    • 5438097 fix compilation issues
    • a78e7e8 fix contiguous file checker bug
    • cbd1963 fix corrupted memory issue
    • 0855181 fix database merge, update tests
    • 24e52dc fix file prefix
    • 004cf40 fix generate.sh, generate the dummy RPC proto def
    • 6e7901c fix get_last not found return values
    • 06eb365 fix get_last not found return values
    • e916b5d fix go generate ./...
    • f4bd406 fix json decoding of special keys
    • 1ed793e fix major memory leak and bump bstream
    • 2483911 fix manifest to proto
    • 294d1f8 fix manifest_tests
    • dc7e415 fix module sort fix logs short name and add better logging
    • 32abe53 fix native code entry point
    • f5a680f fix pipeline logger module name
    • 27f6869 fix pipeline, use proto everywhere
    • b5101e8 fix pprof
    • bdfbdef fix return handler func signature, minor refactoring and fix some compilation issues
    • 4d6c278 fix rpc eth_call to return output data not pointer
    • 47fc934 fix runtime using protobuf
    • 261492c fix state filenames
    • ef5a2e6 fix stores map not populated correctly
    • d540ea7 fix test
    • dd277ec fix test and optimize start block computation
    • 1b3b88d fix test sorry!
    • 67cecdd fix tooling logger, tooling command usage
    • 69231f1 fix update policy
    • 5809a92 fix update policy
    • 5d03e5a fix waiter tests
    • b606d30 fix warnings on Sprintf
    • 52dba7f fix/generate transform proto
    • cb0b746 fixed tests
    • 1e7c1ea fixed tests agains
    • 1235079 fixed tests agains after stepd broke everything!
    • 247e6c9 gitignore
    • 7e86d8d go mod stuff
    • 0c42689 go.sum
    • 38bf5a9 graph node return handler wip
    • 1c56c27 graph: fix sorting
    • ea37811 graph: remove group methods for now, fix tests
    • c721a33 handler: fix yet another goroutine variable scope issue
    • a8bd20c local run refactoring
    • 74c184a make test compile but still failing new logger module back by zap log
    • 37911dd manifest graph: integrate and fix tests
    • da96e7e manifest graph: new struct using graph lib
    • 2470ac6 manifest tests
    • 12aff7c manifest: fix graph ancestors method
    • a12f60c manifest: fix more tests
    • 5c768f2 manifest: fix tests
    • 57cd407 manifest: fix yaml tag for ProtoType
    • 470d988 manifest: modulegraph ModulesDownTo and StoresDownTo now take a list of modules as parameter
    • 2aa9b73 manifest: remove connected requirement for graph
    • 38e7c7c manifest: simplify regex
    • f14d9b5 manifest: test new yaml
    • 59c92c2 manifest: use regex to validate module names
    • 97a7085 merge deleted prefixes, prevent the use of keys with prefix 0xFF, and of 1 characters keys.
    • 20fee58 module hash as folder instead of file prefix
    • 2b29cab module hash now use in state file naming
    • fc6b9ab module start block calculation. wip
    • 27a8cb6 move back engine and store creation into module creation
    • 3f9c811 move code type to modules
    • c658bf3 move database change from PCS to substreams proto
    • 14570cc move database change substreams proto instead of transform
    • 1929b15 move duplicate code from buildNative and buildWasm to Build func
    • 0e87839 move substream crate to its own git repo
    • a1177d9 now use block payload instead of unmarshal and re marshall it
    • 4df49f2 now used string to serialize sum_big_int to heap
    • 8644771 only use waiter when in partial mode.
    • f958d5e partial mode, waiting for parent stores, refactor of store
    • 60c441b partial mode: do not write other stores. save merge result as full image
    • 1b6a4a6 partial mode: wait for kv files of parallel processes
    • 7d88a32 passing in indirect value of rv in applyTableChange and adding more logging information for clarity
    • 7adc8bf pb generate
    • 1ad428f pbsubstreams.StateDelta
    • 4ebd14b pbsubstreams: add DatabaseChanges squash
    • 6b5b39a pipeline: fix goroutine loop bug
    • 3a6c50f pipeline: fix goroutine scope bug
    • f8cc85c pipeline: handle error to WriteState in handler func
    • 841686c pipeline: write full state after waiting for KV
    • a53620b prevent failure on start-block == moduleStartBlock
    • f74d754 progress tracker
    • 0c10555 progress tracker refactorings
    • 605b80a prostgres store
    • 8d481ae proto::decode(...) doesn't take ownership of the message to decode
    • d9bfd21 re added HandlerFunc panic recovery code.
    • dc8205b refactor global i
    • 382efc4 remote command
    • eee6574 remote go mod replace
    • 764ae1c remove StoreFactory, FactoryInterface, TestSore
    • d561d7b remove all panic
    • a0ed73b remove all runtime.OSThreadLock
    • 8b91e2e remove comment dead code
    • cd532f8 remove commented code
    • ecacc02 remove double module instantiation
    • 38da580 remove old todo
    • 82775e0 remove p flag for partial
    • 1b80fda remove println from wasm print
    • e1740a2 remove pub mod memory and only using memory crate
    • c3f9c2b remove requestedBlockNum from build functions params
    • 98da76e remove unused zap-box package.
    • 20c1277 remove viper, long live cobra, fix start-block
    • 2f00f51 removing call to instance.Close() that call vmInstance.Close(). It may have caused this error ``` double free or corruption (fasttop) SIGABRT: abort PC=0x7fd26003303b m=18 sigcode=18446744073709551610 signal arrived during cgo execution
    • 1fd2556 removing unused variables
    • d11e35c rename manif manifest
    • aef848e rename some proto enum constants
    • 8a9682e rename some stuff
    • a9ecbd0 rename some variables and method function
    • f1c1156 renaming data to types and creating new entity when calling get interface
    • a1ccbb8 replace force low state by a pipeline option
    • d261afd resolve merge value types todo
    • 29986e0 return buffer in proto and forget about it in the output to be able to properly write data once calling extern output function
    • 97c19e2 return error if the proto marshall fails on the return handler
    • 7a54d74 return pointer when eth_call
    • 2efd49d rpc cache cleanup and change logging interval
    • 1595170 rpc cache refactoring and improvements.
    • 0478217 rpc cache refactoring, move cache to eth-go
    • 7ac8b11 rpc cache: always save file to disk
    • 82448e0 rpc cache: method cleanup
    • 514db39 rpc cache: remove irrevelant tests
    • 3cfd5ff rpc: fix unit test to remove file system
    • d5e47ef save store state every 10000 blocks
    • d7978f0 setting precision to 100
    • 9bec0c5 small signature refactoring
    • cfe63da small tools to dump state
    • ba38d72 some comments for eduard
    • 37efd41 start block is not optional anymore. added tests for start block computation. start block has max uint64 value by default
    • 6ff345c state io updates
    • 64552c0 state: StoreBlock fixes
    • c47eaa1 state: add logging
    • d20fcb4 state: add squash
    • ccf3b35 state: block numbers in file names are zero-padded for readability
    • 49a8f7c state: cleanup
    • d0a30bc state: fix kv file regex
    • d463a3e state: fix waiter tests
    • bf78bf4 state: implement all setter interfaces
    • c4a08e1 state: implement merge strategies
    • 7572492 state: more data stored in special keys on disk. added command to merge partial files in store.
    • 1c10fa6 state: more logging
    • d4b537d state: store update strategy and value type in special keys
    • b81571b state: update ignore merge policy
    • dc008c6 state_get_last state_get_first now write to output ptr
    • 152f0cf store are now created in build func instead of SynchronizeStore
    • eccf277 store: include module start block in full kv state file
    • 6bbabf6 strconv ParseFloat bitsize from 100 to 64
    • b4b0bd4 substream crate code build tooling to test
    • e6bee7b synchronize is now using pipeline stores list
    • 070b302 test commit
    • 8753d8e testing mono repos
    • 32d37f0 testing mono repos take 2. with compilation error
    • 252e469 tidy up
    • 475d7f3 tools: added merge and cleanup tools. remove old command in cli
    • 5b26231 transforms: some more WIP in Run() cmd
    • 3184b3a update runtime configurations
    • 0cf5039 use switch instead of if statement to check module kind.
    • ff1cc7a waiter refactor
    • 534f33d waiter: do not wait if target block less than or equal to module start block
    • 5ab6457 waiter: fix file prefix in wait func
    • 7edd264 wip
    • da94ab4 wip
    • 2f6adbf wip for alex
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(217 bytes)
    substreams_0.0.5-beta_Darwin_arm64.tar.gz(8.21 MB)
    substreams_0.0.5-beta_Darwin_x86_64.tar.gz(8.45 MB)
Owner
StreamingFast
StreamingFast is a protocol infrastructure company that provides a massively scalable architecture for streaming blockchain data
StreamingFast
Blockchain-go - A repository that houses a blockchain implemented in Go

blockchain-go This is a repository that houses a blockchain implemented in Go. F

Onyeka 1 May 1, 2022
Community-run technology powering the cryptocurrency, and decentralized applications on TrustFi Network

Go TrustFi-Ethereum Official Golang implementation of the TrustFi-Ethereum protocol. Automated builds are available for stable releases and the unstab

TrustFi Network 786 May 26, 2021
A blockchain-based demo that shows an alternative strategy for ensuring data and log integrity on aircraft

A blockchain-based demo that shows an alternative strategy for ensuring data and log integrity on aircraft. (Coded in less than 24 hours for GunnHack)

sckzor 0 Feb 6, 2022
run ABI encoded data against the ethereum blockchain

Run EVM code against a database at a certain block height - Note You can't run this against a running geth node - because that would share the db and

Edgar Aroutiounian 60 Nov 11, 2021
A web-based demonstration of blockchain concepts.

Blockchain Demo A web-based demonstration of blockchain concepts. This is a very basic visual introduction to the concepts behind a blockchain. We int

Anders Brownworth 4.6k Aug 8, 2022
Go language implementation of a blockchain based on the BDLS BFT protocol. The implementation was adapted from Ethereum and Sperax implementation

BDLS protocol based PoS Blockchain Most functionalities of this client is similar to the Ethereum golang implementation. If you do not find your quest

Yongge Wang 0 Jan 1, 2022
A phoenix Chain client based on the go-ethereum fork,the new PoA consensus engine is based on the VRF algorithm.

Phoenix Official Golang implementation of the Phoenix protocol. !!!The current version is for testing and developing purposes only!!! Building the sou

g_master 14 Apr 28, 2022
Streaming Fast on Ethereum

Stream Ethereum data like there's no tomorrow

StreamingFast 69 Jul 9, 2022
DERO Homomorphic Encryption Blockchain Protocol

Homomorphic encryption is a form of encryption allowing one to perform calculations on encrypted data without decrypting it first. The result of the computation is in an encrypted form, when decrypted the output is the same as if the operations had been performed on the unencrypted data.

null 98 Aug 10, 2022
A simplified blockchain implementation in Golang

A simplified blockchain implementation in Golang

Ivan Kuznetsov 3.5k Aug 10, 2022
DERO: Secure, Anonymous Blockchain with Smart Contracts. Subscribe to Dero announcements by sending mail to [email protected] with subject: subscribe announcements

Welcome to the Dero Project DERO News Forum Wiki Explorer Source Twitter Discord Github Stats WebWallet Medium Table of Contents ABOUT DERO PROJECT DE

null 274 May 22, 2022
Go module for the Cardano Blockchain

cardano-go cardano-go is both a library for creating go applicactions that interact with the Cardano Blockchain as well as a CLI to manage Cardano Wal

Alonso Villegas 30 Aug 10, 2022
chia-blockchain some function implement in golang

gochia chia-blockchain some function implement in golang Package bls-signatures implement blspy Usage? Now we can use it to generate plot memo and id,

Kure 8 May 27, 2022
Frontier Chain is a blockchain application built using Cosmos SDK and Tendermint.

Frontier Chain Frontier Chain is a blockchain application built using Cosmos SDK and Tendermint. Setup Initialize the blockchain with one validator no

Frontier 12 Jul 12, 2022
Implementing blockchain using Golang ✔️

Implementing blockchain using Golang ✔️ Keys The Blockchain uses ECDSA (224 bits) keys.

null 7 May 24, 2022
Gochain is a Blockchain written in go

gochain gochain is a proof-of-work blockchain written in go. Features Proof-Of-Work Persistence CLI Transactions Addresses Merkle Tree Network How to

Jonas Ampferl 8 Jul 14, 2022
LINE Financial Blockchain forked from gaia

LFB(LINE Financial Blockchain) This repository hosts LFB(LINE Financial Blockchain). This repository is forked from gaia at 2021-03-15. LFB is a mainn

LINE 31 Jun 27, 2022
utreexo blockchain skeleton

sunyata sunyata is a blockchain skeleton. It implements a minimally-functional proof-of-work blockchain, including consensus algorithms, p2p networkin

The Sia Foundation 18 May 24, 2022
OmniFlix Hub is a blockchain built using Cosmos SDK and Tendermint and created with Starport.

OmniFlix Hub is the root chain of the OmniFlix Network. Sovereign chains and DAOs connect to the OmniFlix Hub to manage their web2 & web3 media operations (mint, manage, distribute & monetize) as well as community interactions.

OmniFlix Network 34 Jun 27, 2022