Ethereum on StreamingFast

Related tags

Cryptography sf-ethereum
Overview

Ethereum on StreamingFast

reference License

Requirements (clone repos, build stuff...)

Install Geth

git clone [email protected]:streamingfast/go-ethereum.git
cd go-ethereum
git checkout release/geth-1.10.x-dm
go install ./cmd/geth
go install ./cmd/bootnode

Install sfeth

git clone [email protected]:streamingfast/sf-ethereum.git
cd sf-ethereum
go install ./cmd/sfeth

Quickstart, connecting to an existing chain

  • Start from a clean folder

  • Create a file named sf.yaml and put the following content:

    start:
args:
- merger
- firehose
- mindreader-node
- relayer
flags:
  common-chain-id: "1"
  common-network-id: "1"
  mindreader-node-bootstrap-data-url: ./mindreader/genesis.json
  mindreader-node-log-to-zap: false
  mindreader-node-arguments: "+--bootnodes=enode://<enode1>@<ip>:<port>,enode://<enode2>@<ip>:<port>"

    Note Up to date boot nodes info for Geth supported network(s) can be found here.

  • Create a folder mindreader

  • Copy the genesis.json file of the chain into the mindreader folder.

    Note It's possible to use geth dumpgenesis to dump actual genesis file to disk

    • Mainnet - geth --mainnet dumpgenesis > ./mindreader/genesis.json
    • Ropsten - geth --ropsten dumpgenesis > ./mindreader/genesis.json
    • Goerli - geth --goerli dumpgenesis > ./mindreader/genesis.json
    • Rinkeby - geth --rinkeby dumpgenesis > ./mindreader/genesis.json

  • sfeth start -vv

    Note It's recommended to launch with -vv the first time to more easily see what's happening under the hood.

  • Wait around a minute leaving enough time for the Geth process to start the syncing process. You should then have some merged blocks under ./sf-data/storage/merged-blocks. You should also be able to test that Firehose is able to stream some blocks to you.

    grpcurl -insecure -import-path ../proto -import-path ../proto-ethereum -proto dfuse/ethereum/codec/v1/codec.proto -proto dfuse/bstream/v1/bstream.proto -d '{"start_block_num": -1}' 127.0.0.1:13042 dfuse.bstream.v1.BlockStreamV2.Blocks

    Note You will need to have grpcurl and a clone of both https://github.com/streamingfast/proto and https://github.com/streamingfast/proto-ethereum, we assume they are sibling of the folder you are currently in, adjust -import-path ... flags in the command above to where the files are located.

Contributing

Issues and PR in this repo related strictly to the Ethereum on StreamingFast.

Report any protocol-specific issues in their respective repositories

Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.

This codebase uses unit tests extensively, please write and run tests.

License

Apache 2.0

Comments
  • Merger stuck at block 31000

    Merger stuck at block 31000

    It seems that my merger is now stuck at block 31000. I see the following log line emitted over and over

    firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"WARNING","timestamp":"2022-10-21T15:44:42.420585258-04:00","logger":"bstream","message":"too many consecutive unlinkable blocks","block":"#109903 (b4a118481e568b1c)","consecutive_unlinkable_blocks":77100,"last_block_sent":"#31272 (00689cb504900f40)","logging.googleapis.com/labels":{}}

    and here's a more complete sample of log lines:

    firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:36.322716469-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"/data/firehose/storage/merged-blocks/0000031000.dbin.zst","base_filename":"0000031000","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:37.068411556-04:00","logger":"reader.geth","message":"forkchoice requested sync to new head    number=15,798,650 hash=d179b5..c50f7d","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:40.322901399-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"/data/firehose/storage/merged-blocks/0000031000.dbin.zst","base_filename":"0000031000","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:40.834577546-04:00","logger":"reader.geth","message":"imported new chain segment               blocks=71   txs=389  mgas=8.229  elapsed=8.235s    mgasps=0.999  number=369,258    hash=f8be77..e6c618 age=7y1mo2w  dirty=5.43MiB","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"WARNING","timestamp":"2022-10-21T15:44:42.420585258-04:00","logger":"bstream","message":"too many consecutive unlinkable blocks","block":"#109903 (b4a118481e568b1c)","consecutive_unlinkable_blocks":77100,"last_block_sent":"#31272 (00689cb504900f40)","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:44.323103322-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"/data/firehose/storage/merged-blocks/0000031000.dbin.zst","base_filename":"0000031000","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:48.323232598-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"/data/firehose/storage/merged-blocks/0000031000.dbin.zst","base_filename":"0000031000","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:48.996961958-04:00","logger":"reader.geth","message":"imported new chain segment               blocks=804  txs=708  mgas=15.726 elapsed=8.115s    mgasps=1.938  number=370,062    hash=266bdd..3b7689 age=7y1mo2w  dirty=4.67MiB","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:49.077330751-04:00","logger":"reader.geth","message":"forkchoice requested sync to new head    number=15,798,651 hash=31b649..2c1d7c","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"WARNING","timestamp":"2022-10-21T15:44:50.162217455-04:00","logger":"bstream","message":"too many consecutive unlinkable blocks","block":"#110003 (7b770ad57aa913a8)","consecutive_unlinkable_blocks":77200,"last_block_sent":"#31272 (00689cb504900f40)","logging.googleapis.com/labels":{}}

    Are the unlinkable blocks an issue? If so, how can I resolve them?

    opened by paymog 8
  • Default command line uses obsolete arguments to geth

    Default command line uses obsolete arguments to geth 

    (mindreader-node) WARN [11-26|15:17:46.583] The flag --rpc is deprecated and will be removed June 2021, please use --http  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpcaddr is deprecated and will be removed June 2021, please use --http.addr  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpcport is deprecated and will be removed June 2021, please use --http.port  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpcapi is deprecated and will be removed June 2021, please use --http.api  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpcvhosts is deprecated and will be removed June 2021, please use --http.vhosts  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable  (log_plugin/to_zap_log_plugin.go:131)
    opened by matthewdarwin 6
  • Mindreader error on startup

    Mindreader error on startup "unsupported log line"

    I am trying to start the sfeth process but it shutdown immediately due to a mindreader error.

    reading from console logs {"error": "unsupported log line: \"INIT 2.0 geth 1.10.21-fh2-86d99626\""}

    2022-08-07T00:29:19.460Z (sfeth) starting with config file './sf.yaml'
2022-08-07T00:29:19.460Z (sfeth) starting atomic level switcher {"listen_addr": "localhost:1065"}
2022-08-07T00:29:19.460Z (sfeth) launching applications: firehose,merger,mindreader-node,relayer
2022-08-07T00:29:20.503Z (mindreader.geth) initializing deep mind 
2022-08-07T00:29:20.511Z (mindreader.geth) deep mind initialized                    enabled=true sync_instrumentation_enabled=true mining_enabled=false block_progress_enabled=false compaction_disabled=false archive_blocks_to_keep=0 genesis_provenance="Geth Default"
2022-08-07T00:29:20.511Z (mindreader) reading from console logs {"error": "unsupported log line: \"INIT 2.0 geth 1.10.21-fh2-86d99626\""}
2022-08-07T00:29:20.512Z (sfeth) app mindreader-node triggered clean shutdown
2022-08-07T00:29:20.512Z (mindreader) {"status": {"Cmd":"./geth","PID":3901,"Exit":-1,"Error":{},"StartTs":1659832160470126042,"StopTs":1659832160512767430,"Runtime":0.042641391,"Stdout":null,"Stderr":null}} command terminated with non-zero status, last log lines:
<None>
2022-08-07T00:29:20.512Z (sfeth) application mindreader-node triggered a clean shutdown, quitting
2022-08-07T00:29:20.512Z (sfeth) waiting for all apps termination...
2022-08-07T00:29:20.513Z (sfeth) all apps terminated gracefully
2022-08-07T00:29:20.513Z (sfeth) goodbye
    opened by rphansen91 5
  • merger should delete old files upon detection

    merger should delete old files upon detection

    Merger keeps finding new files that are old, but it doesn't delete them for some reason.

    Re: https://github.com/streamingfast/sf-ethereum/issues/10

    It seems to wait until it can merge a block, then delete the old files. But it might never merge a block because there are too many old files.

    opened by matthewdarwin 5
  • it sometimes takes mindreader a while to figure out geth is complete on shutdown

    it sometimes takes mindreader a while to figure out geth is complete on shutdown

    image

    This goes on until systemd reaches the programmed timeout and hard kills it. I've seen this many, many times.

    Seen both on BSC and ETH.

    At least it is a clean shutdown so geth will start properly next time.

    Config for BSC as example:

    start:
  args:
  - mindreader-node
  flags:
    verbose: 2
    log-to-file: false
    common-chain-id: "56"
    common-network-id: "56"
    common-blocks-store-url: s3://ceph/.....
    common-oneblock-store-url: s3://ceph/.....
    mindreader-node-grpc-listen-addr: :9000
    mindreader-node-oneblock-suffix: servername
    mindreader-node-path: /usr/bin/geth
    mindreader-node-data-dir: /var/lib/geth
    mindreader-node-arguments: "+--port=30300 --cache=16384 --snapshot=false"
    mindreader-node-merge-and-store-directly: false
    mindreader-node-bootstrap-data-url: /etc/dfuse/genesis.json

    This is not reproducible on demand. I'll try to note when it happens going forward.

    opened by matthewdarwin 5
  • merger: how to increase added_files_count

    merger: how to increase added_files_count

    The added_files_count never goes past 2000. How to increase it?

    2022-01-10T23:01:24.787Z (merger) retrieved list of files (merger/merger.go:179){"too_old_files_count": 0, "added_files_count": 2000}
    opened by matthewdarwin 4
  • error: failed storing block in archiver, shutting down. You will need to reprocess over this range to get this block (mindreader/mindreader.go:275){

    error: failed storing block in archiver, shutting down. You will need to reprocess over this range to get this block (mindreader/mindreader.go:275){"error": "blocks non contiguous, expectedBlock: 5573454, got block: 5573455"...

    Ran into a problem:

    error: failed storing block in archiver, shutting down. You will need to reprocess over this range to get this block (mindreader/mindreader.go:275){"error": "blocks non contiguous, expectedBlock: 5573454, got block: 5573455"...

    image

    2 issues here:

    1. why does mindreader fail to read the correct block,
    2. why does the entire process not just exit? Instead we just left with a partially working system. no firehose blocks are generated.

    Other than reading logs, what is the correct way to detect this problem?

    And how to recover? Restarting mindreader just gets stuck again at the same spot. I don't care about the hole in merged blocks. Another mindreader (running on another server) will fill it in eventually.

    opened by matthewdarwin 4
  • Bad block breaks geth

    Bad block breaks geth

    I've started running firehose and I just ran into the following log lines which seems to have broken geth - it's now reporting that a Disk Quota has been been exceeded:

    firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:24.478946191-04:00","logger":"reader.geth","message":"beacon chain gapped                      head=15,869,849 newHead=15,869,855","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:24.478990291-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:24.477] Failed to clean stale beacon headers     err=\"filled header below beacon header tail: 6158489 < 15869849\"","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:24.693057822-04:00","logger":"reader.geth","message":"unindexed transactions                   blocks=439 txs=55767 tail=6,158,362 elapsed=227.838ms","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:24.880649279-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:25.332971157-04:00","logger":"reader.geth","message":"syncing beacon headers                   downloaded=1540 left=6,158,488 eta=550h18m24.979s","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:25.382861547-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:25.518788928-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:25.518] Expired request does not exist           peer=e053914017a506772cce70913eac44cc9b0e4378adc138f6f11b7c943fe5acbf","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:25.884820224-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:25.955989251-04:00","logger":"reader.geth","message":"downloader queue stats                   receiptTasks=0 blockTasks=34168 itemSize=27.24KiB throttle=8192","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.036932293-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.036] Unhandled trie error: missing trie node 4cdf3958a41d42a048cb37085d0e2450f3a92074a99d446ab4dd7b48cfd190ab (path 03070b030e0c04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.212740567-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.212] Unhandled trie error: missing trie node 164b4f0699da01839785857cce9e614559afc9a569c10b3e82254db1abb7d16b (path 09090b0f0d04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.259208341-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.259] Unhandled trie error: missing trie node 164b4f0699da01839785857cce9e614559afc9a569c10b3e82254db1abb7d16b (path 09090b0f0d04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.268408351-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.268] Unhandled trie error: missing trie node 7217e48235c2c5bed47a4bc25418e952db40c64433424ea1f99b44669000b375 (path 09090e00010900) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.28229091-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.282] Unhandled trie error: missing trie node 7217e48235c2c5bed47a4bc25418e952db40c64433424ea1f99b44669000b375 (path 09090e00010900) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.287387822-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.283] Unhandled trie error: missing trie node ca4cbcd5f6e09f53c448cc68a9b452ce669a25a9eed0f22a79a4e07d4a07bae3 (path 07010e090b0f08) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.287431338-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.284] Unhandled trie error: missing trie node ca4cbcd5f6e09f53c448cc68a9b452ce669a25a9eed0f22a79a4e07d4a07bae3 (path 07010e090b0f08) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30427221-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.304] Unhandled trie error: missing trie node b60717f155b92939dc3387492394afd09912678a26923af5d2c4e5e315c913f2 (path 04070100050e03) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30760353-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.306] Unhandled trie error: missing trie node b60717f155b92939dc3387492394afd09912678a26923af5d2c4e5e315c913f2 (path 04070100050e03) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30780068-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.307] Unhandled trie error: missing trie node 0936e560feba21cfd5ca6686193ede797b4c4a1818436703888696343c38b086 (path 0c09040909020b) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30991422-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.309] Unhandled trie error: missing trie node 0936e560feba21cfd5ca6686193ede797b4c4a1818436703888696343c38b086 (path 0c09040909020b) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.311182544-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.311] Unhandled trie error: missing trie node da95e202db5dadb272aba88d218c26038af5e437514fa80e3f116f60d50cc4f5 (path 0c0a00030108) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.312121839-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.312] Unhandled trie error: missing trie node da95e202db5dadb272aba88d218c26038af5e437514fa80e3f116f60d50cc4f5 (path 0c0a00030108) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:26.393611317-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.40731312-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.407] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.408046113-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.407] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.408799923-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.408] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose FAILED trx "a9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927" at block 6158490 (hash unavailable, probably forked): insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515 2472
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410209421-04:00","logger":"reader.geth","message":"skip duplicated bad block                number=6,158,490  hash=b6e729..3720d4","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.410320327-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.410] ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410359297-04:00","logger":"reader.geth","message":"########## BAD BLOCK #########","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410375414-04:00","logger":"reader.geth","message":"chain config: Chain ID:  1 (mainnet)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410387081-04:00","logger":"reader.geth","message":"consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410399144-04:00","logger":"reader.geth","message":"pre-Merge hard forks:","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41041168-04:00","logger":"reader.geth","message":" - Homestead:                   1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410424045-04:00","logger":"reader.geth","message":" - DAO Fork:                    1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410451966-04:00","logger":"reader.geth","message":" - Tangerine Whistle (EIP 150): 2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410467232-04:00","logger":"reader.geth","message":" - Spurious Dragon/1 (EIP 155): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41047902-04:00","logger":"reader.geth","message":" - Spurious Dragon/2 (EIP 158): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410490538-04:00","logger":"reader.geth","message":" - Byzantium:                   4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410502395-04:00","logger":"reader.geth","message":" - Constantinople:              7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410515128-04:00","logger":"reader.geth","message":" - Petersburg:                  7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410528678-04:00","logger":"reader.geth","message":" - Istanbul:                    9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410545419-04:00","logger":"reader.geth","message":" - Muir Glacier:                9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41057709-04:00","logger":"reader.geth","message":" - Berlin:                      12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410593931-04:00","logger":"reader.geth","message":" - London:                      12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410607918-04:00","logger":"reader.geth","message":" - Arrow Glacier:               13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.4106357-04:00","logger":"reader.geth","message":" - Gray Glacier:                15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41065043-04:00","logger":"reader.geth","message":"merge configured:","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410661111-04:00","logger":"reader.geth","message":" - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41067195-04:00","logger":"reader.geth","message":" - Network known to be merged: true","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410682275-04:00","logger":"reader.geth","message":" - Total terminal difficulty:  58750000000000000000000","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410693105-04:00","logger":"reader.geth","message":" - Merge netsplit block:       <nil>","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410704011-04:00","logger":"reader.geth","message":"number: 6158490","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41072913-04:00","logger":"reader.geth","message":"hash: 0xb6e729d033a32968a21450a3408c8b5ab4566e0c5e1bd112017519d21b3720d4","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410747068-04:00","logger":"reader.geth","message":"error: could not apply tx 113 [0xa9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927]: insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410762094-04:00","logger":"reader.geth","message":"##############################","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410772044-04:00","logger":"reader.geth","message":" ","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.410808153-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.410] Beacon backfilling failed                err=\"retrieved hash chain is invalid: could not apply tx 113 [0xa9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927]: insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515\"","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.874415245-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"merged-blocks/0006158300.dbin.zst","base_filename":"0006158300","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:26.896155225-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:27.399447259-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:27.901626774-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}

    Is this a known issue? Could it be that my geth instance isn't syncing properly? Might the bad block be a red herring for the disk quota error?

    opened by paymog 3
  • mindreader provides --port 30305 is provided even if overridden

    mindreader provides --port 30305 is provided even if overridden

    I have mindreader-node-arguments: "+--port=30301 --cache=2048"

    ps shows me

    /usr/bin/bsc-geth --networkid=56 --datadir=/var/lib/geth --ipcpath=/var/lib/dfuse/sf/ipc --port=30305 --rpc --rpcapi=admin,debug,eth,net,web3 --rpcport=8547 --rpcaddr=0.0.0.0 --rpcvhosts=* --nousb --firehose-deep-mind --port=30301 --cache=2048

    Notice there are two --port arguments. It seems to work by taking the last one, but would like to avoid passing the first one anyway.

    opened by matthewdarwin 3
  • How to consume the firehose block stream?

    How to consume the firehose block stream?

    I've read through the firehose docs and it's not clear how the block stream produced by firehouse should be consumed. Does the gRPC server provide an interface that is compatible with the eth JSON-RPC interface? Does Firehose integrate tightly with graph node (considering it's mentioned in the graph node docs)? How should the grpc server be started, in the startup YAML or as a subcommand?

    opened by paymog 2
  • reader libNumber never moves forward when replaying old chain data

    reader libNumber never moves forward when replaying old chain data

    libNumber never moves forward

    $ grpcurl -plaintext -d '{}' 'localhost:9000' 'sf.headinfo.v1.HeadInfo.GetHeadInfo'
{
  "libNum": "12660716",
  "headNum": "12715300",
  "headID": "1d39f0036deb2770748da57717bd7dc52d2244b524bc0881fb57cb5f0dcea37c",
  "headTime": "2022-08-03T14:50:00Z"
}

    check later:

    $ grpcurl -plaintext -d '{}' 'localhost:9000' 'sf.headinfo.v1.HeadInfo.GetHeadInfo'
{
  "libNum": "12660716",
  "headNum": "12747578",
  "headID": "b869c6f610588ed5618f24cb1e9d8beff6451d85bfec48950700f509092046e5",
  "headTime": "2022-08-08T21:45:12Z"
}
    opened by matthewdarwin 2
  • Devel sync-mainnet stuck on startup (can't find peers)

    Devel sync-mainnet stuck on startup (can't find peers)

    Hi,

    i'm having an issue with running the example from the devel/sync-mainnet/ folder and all links to your discord seem to be expired in docs so I can't find/join it.

    Lighthouse release used: https://github.com/sigp/lighthouse/releases/tag/v3.3.0 Config used(no changes, just for testing): https://github.com/streamingfast/firehose-ethereum/tree/develop/devel/sync-mainnet

    streamingfast/go-ethereum release used: release/geth-v1.10.26-fh2.1

    fireeth --version
fireeth version dev

    Issue: Basically the geth node can't find peers. It gets stuck in this state for >30minutes and then I terminate it. I checked the security groups in AWS and they all seem reasonably open for the ports it needs. I tried opening the ports completely too to 0.0.0.0/0. Other aws compute with rocketpool and erigon nodes running in the same security group work fine so I don't think it's the network.

    For testing:

    export CHECKPOINT_SYNC_URL=https://sync.invis.tools

    Consensys output: https://pastecode.io/s/djzjdw8i

    Firehose output: https://pastecode.io/s/ym9pfdkk

    opened by uberspot 1
  • Build in docker

    Build in docker

    The build-vanilla workflow currently runs the build on the host machine and then loads it into the docker image. This PR changes it so that the build is performed in a build stage within the dockerfile.

    Blocked by https://github.com/moby/buildkit/issues/1512

    Currently, cache mounts only work in incremental builds (locally) but not in github workflows. Once that's resolved and the build speed is acceptable, this can be considered again.

    opened by fubhy 0
  • merger: publish number of files into prometheus (too_old_files_count)

    merger: publish number of files into prometheus (too_old_files_count) 

    2022-01-10T22:25:23.030Z (merger) retrieved list of files (merger/merger.go:179){"too_old_files_count": 0, "too_old_files_count": 884}
    opened by matthewdarwin 1
Releases(v1.2.2)

  • v1.2.2(Nov 24, 2022)

    Substreams improvements

    Performance

    • Changed cache file format for stores and outputs (faster with vtproto) -- requires removing the existing state files
    • Various improvements to scheduling

    Fixes

    • Fixed eth_call handler not flagging out of gas error as deterministic.
    • Fixed Memory leak in wasmtime
      • Hotfix 'nil pointer' panic when saving uninitialized cache (from unreleased v1.2.1 tag)

    Merger fixes

    • Removed the unused 'previous' one-block in merged-blocks (99 inside bundle:100)
    • Fix: also prevent rare bug of bundling "very old" one-blocks in merged-blocks
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(199 bytes)
    checksums.txt.sig(134 bytes)
    fireeth_1.2.2_linux_x86_64.tar.gz(45.78 MB)
    fireeth_1.2.2_macOS_arm64.tar.gz(28.24 MB)
    fireeth_1.2.2_macOS_x86_64.tar.gz(29.38 MB)

  • v1.2.0(Oct 25, 2022)

    Added

    • Added sf.firehose.v2.Fetch/Block endpoint on firehose, allows fetching single block by num, num+ID or cursor.
    • Added tools firehose-single-block-client to call that new endpoint.

    Changed

    • Renamed tools normalize-merged-blocks to upgrade-merged-blocks.

    Fixed

    • Fixed common-blocks-cache-dir flag's description.
    • Fixed DELEGATECALL's caller (a.k.a from). -> requires upgrade of blocks to version: 3

    Upgrade Procedure

    Assuming that you are running a firehose deployment v1.1.0 writing blocks to folders /v2-oneblock, /v2-forked and /v2, you will deploy a new setup that writes blocks to folders /v3-oneblock, v3-forked and /v3

    This procedure describes an upgrade without any downtime. With proper parallelization, it should be possible to complete this upgrade within a single day.

    1. Launch a new reader with this code, running instrumented geth binary: https://github.com/streamingfast/go-ethereum/releases/tag/geth-v1.10.25-fh2.1 (you can start from a backup that is close to head)
    2. Upgrade your merged-blocks from version: 2 to version: 3 using fireeth tools upgrade-merged-blocks /path/to/v2 /path/to/v3 {start} {stop} (you can run multiple upgrade commands in parallel to cover the whole blocks range)
    3. Create combined indexes from those new blocks with fireeth start combined-index-builder (you can run multiple commands in parallel to fill the block range)
    4. When your merged-blocks have been upgraded and the one-block-files are being produced by the new reader, launch a merger
    5. When the reader, merger and combined-index-builder caught up to live, you can launch the relayer(s), firehose(s)
    6. When the firehoses are ready, you can now switch traffic to them.
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(199 bytes)
    checksums.txt.sig(134 bytes)
    fireeth_1.2.0_linux_x86_64.tar.gz(45.08 MB)
    fireeth_1.2.0_macOS_arm64.tar.gz(27.25 MB)
    fireeth_1.2.0_macOS_x86_64.tar.gz(28.70 MB)

  • v1.1.0(Sep 21, 2022)

    v1.1.0

    Added

    • Added 'SendAllBlockHeaders' param to CombinedFilter transform when we want to prevent skipping blocks but still want to filter out trxs

    Changed

    • Reduced how many times reader read statistics is displayed down to each 30s (previously each 5s) (and re-wrote log to reader node statistics).

    Fixed

    • Fix fireeth tools download-blocks-from-firehose tool that was not working anymore.
    • Simplify forkablehub startup performance cases.
    • Fix relayer detection of a hole in stream blocks (restart on unrecoverable issue).
    • Fix possible panic in hub when calls to one-block store are timing out.
    • Fix merger slow one-block-file deletions when there are more than 10000 of them.
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(199 bytes)
    checksums.txt.sig(134 bytes)
    fireeth_1.1.0_linux_x86_64.tar.gz(44.98 MB)
    fireeth_1.1.0_macOS_arm64.tar.gz(27.19 MB)
    fireeth_1.1.0_macOS_x86_64.tar.gz(28.63 MB)

  • v1.0.0(Sep 2, 2022)

    BREAKING CHANGES

    Project rename

    • The binary name has changed from sfeth to fireeth (aligned with https://firehose.streamingfast.io/references/naming-conventions)
    • The repo name has changed from sf-ethereum to firehose-ethereum

    Ethereum V2 blocks (with fh2-instrumented nodes)

    • This will require reprocessing the chain to produce new blocks
    • Protobuf Block model is now tagged sf.ethereum.type.v2 and contains the following improvements:
      • Fixed Gas Price on dynamic transactions (post-London-fork on ethereum mainnet, EIP-1559)
      • Added "Total Ordering" concept, 'Ordinal' field on all events within a block (trx begin/end, call, log, balance change, etc.)
      • Added TotalDifficulty field to ethereum blocks
      • Fixed wrong transaction status for contract deployments that fail due to out of gas on pre-Homestead transactions (aligned with status reported by chain: SUCCESS -- even if no contract code is set)
      • Added more instrumentation around AccessList and DynamicFee transaction, removed some elements that were useless or could not be derived from other elements in the structure, ex: gasEvents
      • Added support for finalized block numbers (moved outside the proto-ethereum block, to firehose bstream v2 block)
    • There are no more "forked blocks" in the merged-blocks bundles:
      • The merged-blocks are therefore produced only after finality passed (before The Merge, this means after 200 confirmations).
      • One-block-files close to HEAD stay in the one-blocks-store for longer
      • The blocks that do not make it in the merged-blocks (forked out because of a re-org) are uploaded to another store (common-forked-blocks-store-url) and kept there for a while (to allow resolving cursors)

    Firehose V2 Protocol

    • This will require changes in most firehose clients
    • A compatibility layer has been added to still support sf.firehose.v1.Stream/Blocks but only for specific values for 'ForkSteps' in request: 'irreversible' or 'new+undo'
    • The Firehose Blocks protocol is now under sf.firehose.v2 (bumped from sf.firehose.v1).
      • Step type IRREVERSIBLE renamed to FINAL
      • Blocks request now only allows 2 modes regarding steps: NEW,UNDO and FINAL (gated by the final_blocks_only boolean flag)
      • Blocks that are sent out can have the combined step NEW+FINAL to prevent sending the same blocks over and over if they are already final

    Block Indexes

    • Removed the Irreversible indices completely (because the merged-blocks only contain final blocks now)
    • Deprecated the "Call" and "log" indices (xxxxxxxxxx.yyy.calladdrsig.idx and xxxxxxxxxx.yyy.logaddrsig.idx), now replaced by "combined" index
    • Moved out the sfeth tools generate-... command to a new app that can be launched with sfeth start generate-combined-index[,...]

    Flags and environment variables

    • All config via environment variables that started with SFETH_ now starts with FIREETH_
    • All logs now output on stderr instead of stdout like previously
    • Changed config-file default from ./sf.yaml to "", preventing failure without this flag.
    • Renamed common-blocks-store-url to common-merged-blocks-store-url
    • Renamed common-oneblock-store-url to common-one-block-store-url now used by firehose and relayer apps
    • Renamed common-blockstream-addr to common-live-blocks-addr
    • Renamed the mindreader application to reader
    • Renamed all the mindreader-node-* flags to reader-node-*
    • Added common-forked-blocks-store-url flag used by merger and firehose
    • Changed --log-to-file default from true to false
    • Changed default verbosity level: now all loggers are INFO (instead of having most of them to WARN). -v will now activate all DEBUG logs
    • Removed common-block-index-sizes, common-index-store-url
    • Removed merger-state-file, merger-next-exclusive-highest-block-limit, merger-max-one-block-operations-batch-size, merger-one-block-deletion-threads, merger-writers-leeway
    • Added merger-stop-block, merger-prune-forked-blocks-after, merger-time-between-store-pruning
    • Removed mindreader-node-start-block-num, mindreader-node-wait-upload-complete-on-shutdown, mindreader-node-merge-and-store-directly, mindreader-node-merge-threshold-block-age
    • Removed firehose-block-index-sizes,firehose-block-index-sizes, firehose-irreversible-blocks-index-bundle-sizes, firehose-irreversible-blocks-index-url, firehose-realtime-tolerance
    • Removed relayer-buffer-size, relayer-merger-addr, relayer-min-start-offset

    MIGRATION

    Clients

    • If you depend on the proto file, update import "sf/ethereum/type/v1/type.proto" to import "sf/ethereum/type/v2/type.proto"
    • If you depend on the proto file, update all occurrences of sf.ethereum.type.v1.<Something> to sf.ethereum.type.v2.<Something>
    • If you depend on sf-ethereum/types as a library, update all occurrences of github.com/streamingfast/firehose-ethereum/types/pb/sf/ethereum/type/v1 to github.com/streamingfast/firehose-ethereum/types/pb/sf/ethereum/type/v2.

    Server-side

    Deployment

    • The reader requires Firehose-instrumented Geth binary with instrumentation version 2.x (tagged fh2)
    • Because of the changes in the ethereum block protocol, an existing deployment cannot be migrated in-place.
    • You must deploy firehose-ethereum v1.0.0 on a new environment (without any prior block or index data)
    • You can put this new deployment behind a GRPC load-balancer that routes /sf.firehose.v2.Stream/* and /sf.firehose.v1.Stream/* to your different versions.
    • Go through the list of changed "Flags and environment variables" and adjust your deployment accordingly.
      • Determine a (shared) location for your forked-blocks.
      • Make sure that you set the one-block-store and forked-blocks-store correctly on all the apps that now require it.
      • Add the generate-combined-index app to your new deployment instead of the tools command for call/logs indices.
    • If you want to reprocess blocks in batches while you set up a "live" deployment:
      • run your reader node from prior data (ex: from a snapshot)
      • use the --common-first-streamable-block flag to a 100-block-aligned boundary right after where this snapshot starts (use this flag on all apps)
      • perform batch merged-blocks reprocessing jobs
      • when all the blocks are present, set the common-first-streamable-block flag to 0 on your deployment to serve the whole range

    Producing merged-blocks in batch

    • The reader requires Firehose-instrumented Geth binary with instrumentation version 2.x (tagged fh2)
    • The reader does NOT merge block files directly anymore: you need to run it alongside a merger:
      • determine a start and stop block for your reprocessing job, aligned on a 100-blocks boundary right after your Geth data snapshot
      • set --common-first-streamable-block to your start-block
      • set --merger-stop-block to your stop-block
      • set --common-one-block-store-url to a local folder accessible to both merger and mindreader apps
      • set --common-merged-blocks-store-url to the final (ex: remote) folder where you will store your merged-blocks
      • run both apps like this fireeth start reader,merger --...
    • You can run as many batch jobs like this as you like in parallel to produce the merged-blocks, as long as you have data snapshots for Geth that start at this point

    Producing combined block indices in batch

    • Run batch jobs like this: fireeth start generate-combined-index --common-blocks-store-url=/path/to/blocks --common-index-store-url=/path/to/index --combined-index-builder-index-size=10000 --combined-index-builder-start-block=0 [--combined-index-builder-stop-block=10000] --combined-index-builder-grpc-listen-addr=:9000

    Other (non-breaking) changes

    Added tools and apps

    • Added tools firehose-client command with filter/index options
    • Added tools normalize-merged-blocks command to remove forked blocks from merged-blocks files (cannot transform ethereum blocks V1 into V2 because some fields are missing in V1)
    • Added substreams server support in firehose app (alpha) through --substreams-enabled flag

    Various

    • The firehose GRPC endpoint now supports requests that are compressed using gzip or zstd
    • The merger does not expose PreMergedBlocks endpoint over GRPC anymore, only HealthCheck. (relayer does not need to talk to it)
    • Automatically setting the flag --firehose-genesis-file on reader nodes if their reader-node-bootstrap-data-url config value is sets to a genesis.json file.
    • Note to other Firehose implementors: we changed all command line flags to fit the required/optional format referred to here: https://en.wikipedia.org/wiki/Usage_message
    • Added prometheus boolean metric to all apps called 'ready' with label 'app' (firehose, merger, mindreader-node, node, relayer, combined-index-builder)
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(199 bytes)
    checksums.txt.sig(134 bytes)
    fireeth_1.0.0_linux_x86_64.tar.gz(44.91 MB)
    fireeth_1.0.0_macOS_arm64.tar.gz(27.15 MB)
    fireeth_1.0.0_macOS_x86_64.tar.gz(28.58 MB)

  • v0.10.2(Apr 20, 2022)

  • v0.10.1(Apr 13, 2022)

    Changelog

    v0.10.1

    • Fix panic in merger when no prior merged blocks exist

    v0.10.0 (deleted)

    Fixed

    • Fixed an issue where the merger would get stuck when too many (more than 2000) one-block-files were lying around, with block numbers below the current bundle high boundary.
    • Fixed tools check merged-blocks block hole detection behavior on missing ranges (bumped sf-tools)
    • Fixed a deadlock issue related to s3 storage error handling (bumped dstore)
    • Fixed Firehose gRPC listening address over plain text.
    • Fixed automatic merging of files within the mindreader is much more robust then before.

    Changed

    • Renamed common atm 4 flags to blocks-cache: --common-blocks-cache-{enabled|dir|max-recent-entry-bytes|max-entry-by-age-bytes}
    • The default text encoder use to encode log entries now emits the level when coloring is disabled.
    • Default value for flag --mindreader-node-enforce-peers is now "", this has been changed because the default value was useful only in development when running a local node-manager as either the miner or a peering node.
    • Updated all Firehose stack direct dependencies.
    • Updated confusing flag behavior for --common-system-shutdown-signal-delay and its interaction with gRPC connection draining in firehose component sometimes preventing it from shutting down.
    • Reporting an error is if flag merge-threshold-block-age is way too low (< 30s).

    Added

    • Added tools download-from-firehose command to fetch blocks and save them as merged-blocks files locally.
    • Added cloud-gcp:// auth module (bumped dauth)
    • Added block data file caching (called ATM), this is to reduce the memory usage of component keeping block objects in memory.
    • Added transforms: LogFilter, MultiLogFilter, CallToFilter, MultiCallToFilter to only return transaction traces that match logs or called addresses.
    • Added support for irreversibility indexes in firehose to prevent replaying reorgs when streaming old blocks.
    • Added support for log and call indexes to skip old blocks that do not match any transform filter.

    Removed

    • Removed some old components that are not required by Firehose stack directly, the repository is as lean as it ca now.
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(314 bytes)
    sf-ethereum_0.10.1_linux_x86_64.tar.gz(11.64 MB)
    sf-ethereum_0.10.1_macOS_arm64.tar.gz(12.20 MB)
    sf-ethereum_0.10.1_macOS_x86_64.tar.gz(12.07 MB)

  • v0.10.0-rc.5(Apr 7, 2022)

    Changed

    • Renamed common atm 4 flags to blocks-cache: --common-blocks-cache-{enabled|dir|max-recent-entry-bytes|max-entry-by-age-bytes}

    Fixed

    • Fixed tools check merged-blocks block hole detection behavior on missing ranges (bumped sf-tools)
    • Fixed a deadlock issue related to s3 storage error handling (bumped dstore)

    Added

    • Added tools download-from-firehose command to fetch blocks and save them as merged-blocks files locally.
    • Added cloud-gcp:// auth module (bumped dauth)
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(362 bytes)
    sf-ethereum_0.10.0-rc.5_linux_x86_64.tar.gz(11.64 MB)
    sf-ethereum_0.10.0-rc.5_macOS_arm64.tar.gz(12.20 MB)
    sf-ethereum_0.10.0-rc.5_macOS_x86_64.tar.gz(12.07 MB)

  • v0.10.0-rc.4(Apr 4, 2022)

  • v0.10.0-rc.1(Mar 23, 2022)

    Added

    • Added block data file caching (called ATM), this is to reduce the memory usage of component keeping block objects in memory.
    • Added transforms: LogFilter, MultiLogFilter, CallToFilter, MultiCallToFilter to only return transaction traces that match logs or called addresses.
    • Added support for irreversibility indexes in firehose to prevent replaying reorgs when streaming old blocks.
    • Added support for log and call indexes to skip old blocks that do not match any transform filter.

    Changed

    • Updated all Firehose stack direct dependencies.
    • Updated confusing flag behavior for --common-system-shutdown-signal-delay and its interaction with gRPC connection draining in firehose component sometimes preventing it from shutting down.
    • Reporting an error is if flag merge-threshold-block-age is way too low (< 30s).

    Removed

    • Removed some old components that are not required by Firehose stack directly, the repository is as lean as it ca now.

    Fixed

    • Fixed Firehose gRPC listening address over plain text.
    • Fixed automatic merging of files within the mindreader is much more robust then before.

    Full Changelog

    • 3ac8ec4 Add NewEthBlockIndexer() for convenience
    • 6273adf Add TestEthBlockIndexProvider_Matches()
    • ad4e41d Add TestEthBlockIndexProvider_NextMatching()
    • 44eb852 Add TestEthBlockIndexProvider_WithinRange()
    • d21f4ac Add docstrings to EthBlockIndexer
    • 85238a3 Add multi-log filter transform
    • 57c299a Add stubs for BlockIndexProvider
    • d341b4a Added changelog for v0.10.0-rc.1
    • ad161f7 Added git push prior release to ensure GitHub created release creates the correct tag
    • d1f1751 Bump node-manager (moving some backup related stuff in there)
    • 633a782 Extract WriteIndex() and ReadIndex() to bstream
    • 52cd6af Extract chain-agnostic ProcessBlock() logic to bstream
    • dbb5101 Fixed release script when doing a real non dry run
    • 58042c9 Implement Index filter on eth.Address and eth.Hash
    • 2ccd1ae Instantiating ProtocolSpefificModules
    • 6954b53 Merge pull request #24 from streamingfast/stepd/multi-filter
    • c29b0e7 Merge pull request #25 from streamingfast/froch/block-indexer
    • d96cf3a Merge remote-tracking branch 'origin/develop' into froch/block-indexer
    • 52871e4 Remove LogAddressIndex implementations in favor of BlockIndex
    • 4a299ae Removed blockmeta and trxdb from sf-ethereum public
    • 6a2cc3a Removing snapshotter dependency
    • 02e63d7 Replace NewLogAddressIndexProvider() with NewEthBlockIndexProvider()
    • e42aeb5 Replace NewLogAddressIndexer() with NewEthBlockIndexer()
    • 166858e Rework EthBlockIndexProvider to accept a filterFunc
    • 6c3fe6a Sanitize go.mod a bit
    • 7cfb743 Updated bunch of dependencies to get proper logger name
    • 0f2b0f4 Updated dlauncher to latest version
    • 31e4f56 Updated to correct released version
    • af40cee Updated to latest dlauncher
    • ff0fc01 Updated to latest dlauncher and logging library
    • e528270 Updated to latest version of bstream, pbgo, dgrpc, relayer, merger and blockmeta
    • c4b08b0 Updated to latest version of everything
    • da46fb0 Updated to streamingfast/overseer
    • 0583610 Wrap a bstream.transform.BlockIndexer as an EthBlockIndexer
    • 34d9054 account indexing tool to also produce irreversible blocks and start from where it left off
    • bfd49b2 add proper logging of log/call filter transforms
    • fc70c15 adjust accoutn_indexer to start from lowest of irreversible/account index
    • 21aca49 bump bstream for startblock index matching fix
    • f73cd60 bump bstream so firehose always sends start-block even when filtered
    • 321e255 bump bstream to fix firehose cursor on indexed filesource
    • 7b4bbbf bump bstream with my refactors on froch PR
    • add26a0 bump bstream, firehose to fix stop-block bug in indexes
    • 738745b bump deps to fix mindreader merging directly
    • c136aa6 bump node-manager to fix double-compressed oneblockfiles
    • 30bde30 bump node-manager to fix file deletion
    • 75649ed bump node-manager/merger to fix direct merging
    • 878a640 bump node-manager: merge immediately on boundary
    • 29a8fbb change 'transforms' to 'transform' in proto, bump some index libs
    • 24ac3a1 fix confusing flag behavior for grpcShutdownGracePeriod sometimes preventing firehose from shutting down
    • 50787c9 fix logfilter event matching with no topic
    • 35d5311 go mod tidy
    • b3725d9 iadd MultiCallToFilter transform and all that jazz
    • a0e3e44 opiniated failure if merge-threshold-block-age is way too low https://github.com/streamingfast/node-manager/issues/23
    • 443639f remove firehose flag to gen irr.index, bump libs
    • 9465b6b simplify logic in log_address_index
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(329 bytes)
    checksums.txt.sig(134 bytes)
    sf-ethereum_0.10.0-rc.1_linux_x86_64.tar.gz(11.48 MB)
    sf-ethereum_0.10.0-rc.1_macOS_arm64.tar.gz(12.03 MB)
    sf-ethereum_0.10.0-rc.1_macOS_x86_64.tar.gz(11.90 MB)

  • v0.9.0(Feb 16, 2022)

    Changelog

    • 930ded3 Add TestBasicLogFilter_GetIndexProvider()
    • cde9fb5 Add TestLogAddressIndexProvider_LoadRange()
    • 51e6c2a Add TestLogAddressIndexProvider_Matches()
    • da5c7c2 Add TestLogAddressIndexProvider_NextMatching()
    • aae09d7 Add TestLogAddressIndexProvider_WithinRange()
    • 428d1bc Add another case for TestLogAddressIndexProvider_NextMatching()
    • cb30473 Add blockNum as log field in findIndexContaining()
    • 538a4b5 Add docstring to test fixture
    • 3baa7a4 Add guards for GetIndexProvider instantiation
    • c4d7b34 Add missing case for TestLogAddressIndexProvider_NextMatching()
    • e784dd1 Add more details to docstrings
    • b139474 Add stubs for LogAddressIndexProvider
    • 5b8c23a Add support for LogAddressIndex (#17)
    • bfa3f76 Added .vscode to ignore list
    • 4f6db8d Added CI config, added release script and config, updated version printing
    • 5f2901e Added Receipt in TrxTrace when creating a new transaction trace
    • f285ef5 Added missing Burn balance change reason
    • 4efe69e Adjusted instructions
    • 4065b9b Adjusted now removed -rpc flags
    • a2d34c6 Bump snapshotter
    • 6b42571 Bumpd node-manager to latest version
    • 042e1c9 Bumped bstream & firehose to latest version
    • f02ef9f Bumped bstream + firehose
    • fe4994c Bumped dstore to latest version
    • 321ad4b Bumped firehose
    • f092458 Bumped firehose
    • dd5113a Bumped firehose & bstream
    • 05e7e7c Bumped firehose to latest version
    • a4c342d Bumped node-manager and improved logging when running init command
    • d298b78 Bumping to a more recent snapshotter
    • 3338f5e Change method signatures for Matches() and NextMatching()
    • 10caef1 Cherry-pick stepd/wip
    • 7cb85bb Event log transform
    • 70fc4aa First pass - add a LogAddressIndexProvider
    • e28c17a Fix BasicLogFilter and added testing
    • 3acfe9b Fix FindIndexContaining_LoadIndex()
    • 8fef352 Fix blockstream flag
    • 794e2df Fixed --deep-mind flags that are now prefixed with --firehose-deep-mind
    • d11686b Fixed pbcodec.Log#BlockIndex that is wrong when fetched from transactionTraces.calls
    • 1bc6aff Fixed generation of proto files
    • 0496a3d Fixed go.sum
    • 32f88d9 Fixed pb/dfuse not move correctly to pb/sf
    • dbb5101 Fixed release script when doing a real non dry run
    • 6205bad Fixed some small layout issue in motd
    • 9ed057a Flesh-out LogAddressIndexProvider interface methods
    • fb13f31 Implement GetIndexProvider for Transform / BasicLogFilter
    • 2efada2 Implement GetIndexProvider for Transform / BasicLogFilter (#21)
    • 53a4ea6 Increase indexOpsTimeout to 15 seconds
    • 9873294 Initial commit
    • 67bf8df Marshal / Unmarshal a LogAddressIndex from dstore
    • e487c01 Marshal index file and write-out to dstore
    • bb7e801 Merge branch 'develop' into stepd/irreversibility_idx
    • b2b0215 Merge branch 'stepdrename' into develop
    • 6f0aaf1 Merge pull request #14 from streamingfast/stepd/irreversibility_idx
    • acd0e9b Merge pull request #19 from streamingfast/froch/wip
    • 6bc5ef8 Moved check merged blocks logic to a shared library sf-tools
    • 3aeb546 Moved versionning stuff into cli package
    • 799676d Re-added Matches and Call Matches implementation
    • 1a90c1b Re-generated codec protobuf
    • 520bedb Reduce FindIndexContaining_LoadIndex() to a single test case
    • ced9693 Remove NewLogAddressSignatureIndex()
    • ac671a3 Remove deprecated comments relative to IndexProvider interface
    • c1d7b1c Remove vestigial len() check in test fixture
    • 433d127 Renamed deth to codec
    • 269b537 Renamed scripts header
    • f4c71f9 Replace dfuse-io/rpc by streamingfast/rpc
    • 375b7e7 Replacement are now grouped to read better
    • e08381a Rework LogAddressIndex
    • 8f98820 Separate logAddressIndex and logAddressIndexer
    • 859aaff Set require.False(t, done) on NextMatching result
    • 1d6a9ad Try something else to remove snapshotter dependency error
    • 4acfb1a Turn snapshot without compatible eth support into a debug log
    • feab03b Updated logging to latest version
    • cbaec85 Updated merger to latest
    • 1702100 Updated sf-tools for better hole catching in summary printing
    • 2963bbe add 'tools generate-irreversible-index' and parametric bundle-sizes to firehose
    • 3aa7bf4 add NewLogADdressIndex
    • b8a7b14 add draft implementation of log_address_index
    • 3320d36 add logADdressIndex matchingBlocks impl
    • d366261 add log_address_index provider from transform
    • b3b5b55 add status to trxTrace in decoding proto
    • 1cfc5cd add transform/indexes from proto defs
    • 7874ab8 adjust merger params bump bstream, relayer, nodemanager
    • 84e8452 bump bstream and firehose to feature/better_clone_handling
    • b353a6f bump bstream for firehose irrIndex fixes
    • 1b60819 bump bstream for firehose typo
    • e830545 bump bstream for irreversible index creation on first streamable block
    • ea92e86 bump bstream for some cleanups
    • dca87fe bump bstream to feature/better_clone_handling
    • 101fc38 bump bstream to feature/better_clone_handling
    • d827714 bump bstream to feature/better_clone_handling
    • 6b3e98c bump bstream to feature/better_clone_handling
    • 3ddd9a8 bump bstream to feature/better_clone_handling
    • c832acc bump bstream to feature/better_clone_handling
    • 9455ccf bump bstream to feature/better_clone_handling
    • 6f4aa03 bump bstream to feature/better_clone_handling
    • 541cbf9 bump bstream to fix firehose CURSOR issues (cursor without lib)
    • 0f57a42 bump bstream to fix last (incorrectly fixed) typo
    • 64d1b02 bump bstream to fix new memoize/clone, add ATM cache
    • 80da1bd bump bstream to fix/index-perf, adjust logaddr_idx
    • 4b6f143 bump bstream to fix/index-perf, adjust logaddr_idx (#22)
    • d8da678 bump bstream to hotfix atm filesource
    • e570734 bump bstream to merged version of PR
    • 9d1877d bump bstream, fix atm panic
    • cb0ec8b bump bstream, fix some startblocks etc.
    • ac0e774 bump bstream, no ProtoGenesisBlk, fixed cursors
    • ee03862 bump bstream,firehose, add irreversible_blocks_index partial impl.
    • df9c41f bump bstream/firehose after merge
    • d4e2b1e bump bstream/firehose irreversibility_idx
    • 14d6660 bump firehose and bstream to latest develop
    • fb3c9c6 bump firehose for transforms/preproc fix
    • 26c45b0 bump firehose, bstream
    • 100a2a4 bump merger
    • 598b552 bump merger and bstream
    • 65e0b32 bump merger fix toOneBlockFile error handling
    • c66cbf7 bump merger merged file upload fix
    • bb6784b bump merger to drift metrics
    • 53df86d bump merger to fix bootstrap
    • 22f9145 bump merger with better log
    • 1fce895 bump merger with bootstrap from merged files
    • e3f2480 bump merger with fixed bootstrap
    • dde40e9 bump merger with new bootstrap from lib
    • 848d762 bump pbgo and bstream, proto rename from dfuse to sf
    • 53b3c64 continue prototype log address index
    • 299415e fix WithinRange inefficiencies
    • d95a4eb fix log_filter to use Receipt and skip failed logs
    • 64b38bc implement LogAddressIndexProvider.findIndexContaining()
    • 8f8e350 implement LogAddressIndexProvider.loadIndex()
    • 9c1bea3 log in log_address_indexer tool
    • 81bdc9a logAddrIdxProvider fix New() assumptions
    • d166d6d modify log_address_indexer to work with ethblock not bstreamblock
    • 5f4c1dc rebase on develop with transforms
    • 775af98 remove admin,debug rpc on default geth flags, nousb and lachesis references
    • cbfb9ee remove unsued merger flag 'merger-max-fixable-fork'
    • a5c33a2 rename proto index LogAddressMethod to LogAddressSignature
    • 90e77d0 wip, debugging test cases with eth blocks
    • 8095aac work in progress for filter fix, does not compile
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(311 bytes)
    checksums.txt.sig(134 bytes)
    sf-ethereum_0.9.0_linux_x86_64.tar.gz(17.72 MB)
    sf-ethereum_0.9.0_macOS_arm64.tar.gz(18.84 MB)
    sf-ethereum_0.9.0_macOS_x86_64.tar.gz(18.56 MB)
Owner
StreamingFast
StreamingFast is a protocol infrastructure company that provides a massively scalable architecture for streaming blockchain data
StreamingFast
GitHub
Powerful Blockchain streaming data engine, based on StreamingFast Firehose technology.

Substreams - A streaming data engine for The Graph - by StreamingFast DEVELOPER PREVIEW OF SUBSTREAMS Think Fluvio for deterministic blockchain data.

StreamingFast 59 Dec 30, 2022
This plugin enhances Hashicorp Vault Service with cryptographic operations to create, import and sign using different type of keypairs and Ethereum wallets, including signing operation for public ethereum transaction, EEA and Quorum

Quorum Hashicorp Vault plugin The Quorum plugin enhances Hashicorp Vault Service with cryptographic operations under Vault engine, such as: Create and

ConsenSys Software 9 Jan 7, 2023
LEO (Low Ethereum Orbit) is an Ethereum Portal Network client.

LEO LEO (Low Ethereum Orbit) is an Ethereum Portal Network client. What makes LEO different from other Portal Network clients is that it uses libp2p f

Valist, Inc. 10 Apr 19, 2022
Ethereum-vanity-wallet - A fork of https://github.com/meehow/ethereum-vanity-wallet but the key can be exported to a JSON keystore file

ethereum-vanity-wallet See https://github.com/meehow/ethereum-vanity-wallet This version: doesn't display the private key let's you interactively expo

null 0 Jan 2, 2022
Go-ethereum - Official Golang implementation of the Ethereum protocol

Go Ethereum Official Golang implementation of the Ethereum protocol. Automated b

i06 0 Jan 4, 2022
This library aims to make it easier to interact with Ethereum through de Go programming language by adding a layer of abstraction through a new client on top of the go-ethereum library.

Simple ethereum client Simple ethereum client aims to make it easier for the developers to interact with Ethereum through a new layer of abstraction t

Jero 3 May 1, 2022
Huobi Eco Chain client based on the go-ethereum fork

The Huobi Open Platform is a unified infrastructure platform based on the technical, traffic and ecological resources of the Huobi Group, and will be gradually open to the blockchain industry.

null 239 Dec 31, 2022
Streaming Fast on Ethereum

Stream Ethereum data like there's no tomorrow

StreamingFast 73 Dec 15, 2022
a Golang sdk for working with DeFi protocols, and ethereum compatible blockchains

A golang sdk for working with DeFi protocols and general utilities for working with ethereum-compatible blockchains. packages bclient bindings cli con

bonedaddy 74 Dec 15, 2022
run ABI encoded data against the ethereum blockchain

Run EVM code against a database at a certain block height - Note You can't run this against a running geth node - because that would share the db and

Edgar Aroutiounian 60 Nov 11, 2021
Go implementation of Ethereum proof of stake

Prysm: An Ethereum Consensus Implementation Written in Go This is the core repository for Prysm, a Golang implementation of the Ethereum Consensus spe

Prysmatic Labs 2.9k Jan 1, 2023
Ethereum Dapp Go API

Web3 Go Ethereum Dapp Go API, inspired by web3.js. Report Bug · Pull Request Introduction This is the Ethereum Golang API which connects to the Generi

Kyle Liu 15 Nov 29, 2022
A phoenix Chain client based on the go-ethereum fork,the new PoA consensus engine is based on the VRF algorithm.

Phoenix Official Golang implementation of the Phoenix protocol. !!!The current version is for testing and developing purposes only!!! Building the sou

g_master 14 Apr 28, 2022
Evmos is a scalable, high-throughput Proof-of-Stake blockchain that is fully compatible and interoperable with Ethereum.

Evmos Evmos is a scalable, high-throughput Proof-of-Stake blockchain that is fully compatible and interoperable with Ethereum. It's built using the Co

Tharsis 1.4k Dec 31, 2022
Monorepo implementing the Optimistic Ethereum protocol

The Optimism Monorepo TL;DR This is the primary place where Optimism works on stuff related to Optimistic Ethereum. Documentation Extensive documentat

OMG Network 39 Sep 18, 2022
Tool for monitoring your Ethereum clients. Client-agnostic as it queries the standardized JSON-RPC APIs

e7mon Tool for monitoring your Ethereum clients. Client-agnostic as it queries the standardized JSON-RPC APIs. However, the execution client should be

null 31 Dec 20, 2022
Ethereum 2.0 node multiplexer between consensus and execution

The Minority Client Run the minority client! ~Danny Ryan and/or Tim Beiko As of writing, Ethereum has multiple client implementations, but Geth / go-e

Péter Szilágyi 96 Dec 23, 2022
A Commander for Go implementation of official Ethereum Client

Young A Commander for Go implementation of official Ethereum Client by zhong-my. Overview Young Dependencies Young stands on the shoulder of many grea

Zhong MingYang 1 Oct 14, 2021
Ethereum event tracker

Eth-event-tracker Tracker of Ethereum events. Example package main import ( "context" "encoding/binary" "flag" "fmt" "os" "os/signal" "syscall

null 15 Nov 29, 2022