harmony
Reduce Chaos in MemPool
Table of Contents
- Why did you write
harmony? - What do I need to have for installing
harmony? - How do I get
harmonyup & running ? - How do I interact with
harmony?- Checking overall status of mempool
- Catching Any Tx leaving/ joining Mempool
- Catching Tx(s) From
Ain Mempool - Catching Tx(s) To
Ain Mempool - Inspecting tx(s) in pending pool
- Pending For >=
X - Pending For <=
X - Pending From Address
A - Pending To Address
A - Top
XPending Tx(s) - Pending Duplicate Tx(s)
- New Pending Tx(s) [ WebSocket ]
- New Confirmed Tx(s) [ WebSocket ]
- Catch All Pending Pool Changes [ WebSocket ]
- New Pending Tx(s) From Address
A[ WebSocket ] - New Confirmed Tx(s) From Address
A[ WebSocket ] - Catching new pending tx from
A[ WebSocket ] - New Pending Tx(s) To Address
A[ WebSocket ] - New Confirmed Tx(s) To Address
A[ WebSocket ] - Catching new pending tx to
A[ WebSocket ]
- Pending For >=
- Inspecting tx(s) in queued pool
- Queued For >=
X - Queued For <=
X - Queued From Address
A - Queued To Address
A - Top
XQueued Tx(s) - Queued Duplicate Tx(s)
- New Queued Tx(s) [ WebSocket ]
- New Unstuck Tx(s) [ WebSocket ]
- Catch All Queued Pool Changes [ WebSocket ]
- New Queued Tx(s) From Address
A[ WebSocket ] - New Unstuck Tx(s) From Address
A[ WebSocket ] - Catching new queued tx from
A[ WebSocket ] - New Queued Tx(s) To Address
A[ WebSocket ] - New Unstuck Tx(s) To Address
A[ WebSocket ] - Catching new queued tx to
A[ WebSocket ]
- Queued For >=
- Any easy to use test ground for API ?
- Do you have any example(s), showing programmatically querying/ subscribing to GraphQL API ?
Motivation
I discovered Ethereum's MemPool is one of the least explored domains, but not really least important.
Whenever a block is mined & some tx(s) are included in it, it's pretty much, value living at rest, whereas is case of mempool, value is in-flight. A lot of tx(s) are fighting for their space in next block to be mined, where only a few will get their place. But who will get, based or what criteria, it's not very much well defined.
We generally believe giving higher gas price compared to other tx(s) currently present in mempool, gives better chance that block miner will pick this tx during next block mining. But block miner always gets to override that, using custom logic. Also any one can write an automated program for monitoring mempool and replicate tx(s) of their interest with their own address, higher gas price - which may help them in cutting deal faster than original one or benefitting if target smart contract has some security loophole.
During my journey of exploring Ethereum MemPool, I found good initiative from BlockNative in demystifying MemPool. They've built some interesting products on top of mempool.
harmony - Reduce Chaos in MemPool π, aims to become a reliable mempool monitoring engine, while exposing useful functionalities for letting client applications write their monitoring logic seamlessly, with out worrying about underlying details too much
- You can subscribe to listen for tx(s) going to/ from address of interest
- You can catch duplicate nonce tx(s), which of them gets accepted/ dropped
- You can build notification service on top of it
- It will help you in getting better gas price prediction
- It can be used for building real-time charts showing current network traffic
- Many more ...
Prerequisite
- Make sure you've
Go ( >= 1.16),makeinstalled - You need to also have
Redis ( >= 5.x )
Note : Consider setting up Redis instance with password protection
- Get one Ethereum Node up & running, with
txpoolRPC API enabled. You can always use SaaS Ethereum node.
Installation
- For using
harmony, let's first clone this repo
git clone https://github.com/itzmeanjan/harmony.git
- After getting inside
harmony, create.envfile withπ content
cd harmony
touch .env
RPCUrl=https://<rpc-node>
MemPoolPollingPeriod=1000
PendingTxEntryTopic=pending_pool_entry
PendingTxExitTopic=pending_pool_exit
QueuedTxEntryTopic=queued_pool_entry
QueuedTxExitTopic=queued_pool_exit
RedisConnection=tcp
RedisAddress=127.0.0.1:6379
RedisPassword=password
RedisDB=1
ConcurrencyFactor=10
Port=7000
| Environment Variable | Interpretation |
|---|---|
| RPCUrl | txpool RPC API enabled Ethereum Node's URI |
| MemPoolPollingPeriod | RPC node's mempool to be checked every X milliseconds |
| PendingTxEntryTopic | Whenever tx enters pending pool, it'll be published on Redis topic t |
| PendingTxExitTopic | Whenever tx leaves pending pool, it'll be published on Redis topic t |
| QueuedTxEntryTopic | Whenever tx enters queued pool, it'll be published on Redis topic t |
| QueuedTxExitTopic | Whenever tx leaves queued pool, it'll be published on Redis topic t |
| RedisConnection | Communicate with Redis over transport protocol |
| RedisAddress | address:port combination of Redis |
| RedisPassword | Authentication details for talking to Redis. [ Not mandatory ] |
| RedisDB | Redis database to be used. [ By default there're 16 of them ] |
| ConcurrencyFactor | Whenever concurrency can be leveraged, harmony will create worker pool with #-of logical CPUs x ConcurrencyFactor go routines. [ Can be float too ] |
| Port | Starts HTTP server on this port ( > 1024 ) |
- Let's build & run
harmony
make run
Usage
Status of MemPool
For checking current state of mempool, you can issue one HTTP GET request
Method : GET
URL : /v1/stat
curl -s localhost:7000/v1/stat | jq
You'll receive response like
{
"pendingPoolSize": 67,
"queuedPoolSize": 0,
"uptime": "29.214603s",
"networkID": 137
}
| Field | Interpretation |
|---|---|
| pendingPoolSize | Currently these many tx(s) are in pending state i.e. waiting to be picked up by some miner when next block gets mined |
| queuedPoolSize | These tx(s) are stuck, will only be eligible for mining when lower nonce tx(s) of same wallet gets mined |
| uptime | This mempool monitoring engine is alive for last t time unit |
| networkID | The mempool monitoring engine keeps track of mempool of this network |
Catching Any Mempool Changes
Whenever any change in mempool pool happens i.e. tx joins/ leaves pending/ queued pool, subscriber will be notified of those.
- Tx joins queued pool, when it's stuck due to some nonce gap
- It'll leave queued pool, when it's unstuck & lower nonce tx is processed
- Tx joins pending pool, when it's ready to be included in next block [ though might not ]
- Tx leaves pool, when tx it has been included in just mined block
Aforementioned changes generally happen in mempool & using following subscription API lets you capture all of those.
Transport : WebSocket
URL : /v1/graphql
subscription {
memPool{
from
to
gasPrice
pool
pendingFor
queuedFor
}
}
Catching Tx(s) From A in Mempool
When some new tx joins either of queued/ pending pool & that tx is sent from address A, subscriber to be notified of it.
When that tx will leave pool, client to be notified.
Transport : WebSocket
URL : /v1/graphql
subscription {
newTxFromAInMemPool{
from
to
gasPrice
pool
pendingFor
queuedFor
}
}
Catching Tx(s) To A in Mempool
When some new tx joins either of queued/ pending pool & that tx is sent to address A, subscriber to be notified of it.
When that tx will leave pool, client to be notified.
Contract creation tx(s) will have empty
tofield
Transport : WebSocket
URL : /v1/graphql
subscription {
newTxToAInMemPool{
from
to
gasPrice
pool
pendingFor
queuedFor
}
}
Pending Pool
Pending pool inspection related APIs.
Pending for more than X
For listing all tx(s) pending for more than or equals to x time unit, send graphQL query
Method : POST
URL : /v1/graphql
query {
pendingForMoreThan(x: "10s") {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
You'll receive response of form
{
"data": {
"pendingForMoreThan": [
{
"from": "0xdF0692E287A763e5c011cc96Ee402994c6Dd246E",
"gas": "35743",
"gasPrice": "74 Gwei",
"hash": "0x142f95b4615ad31d5435fb979a07405d50b70a2dab2707001cdb04853b75537e",
"input": "0x22c67519000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000001e35",
"nonce": "108",
"to": "0x86935F11C86623deC8a25696E1C19a8659CbF95d",
"value": "0x0",
"v": "0x136",
"r": "0x4becd37941425526e5a1d361a44fd5f911affacaa5526e42e7a20c4a9fb04f90",
"s": "0x3052c55bf6ac67326b4adb92c9ff3288ffe0f0be829b726c2a1cf5b9a58dca5c",
"pendingFor": "10.677797s",
"queuedFor": "0 s",
"pool": "pending"
}
]
}
}
Pending for less than X
For listing all tx(s) pending for less than or equals to x time unit, send graphQL query
Method : POST
URL : /v1/graphql
query {
pendingForLessThan(x: "1m10s") {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Pending from A
For getting a list of all pending tx(s) from specific address, send a graphQL query like
Note : More than one pending tx from same address, denotes those are same nonce tx(s).
Method : POST
URL : /v1/graphql
query {
pendingFrom(addr: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313") {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Pending to A
For getting a list of all pending tx(s) sent to specific address, you can send a graphQL query like
Method : POST
URL : /v1/graphql
query {
pendingTo(addr: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313") {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Top X pending
Top X pending transaction(s), with high gas price
Method : POST
URL : /v1/graphql
query {
topXPendingWithHighGasPrice(x: 10) {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Top X pending transaction(s), with low gas price
Method : POST
URL : /v1/graphql
query {
topXPendingWithLowGasPrice(x: 10) {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Pending Duplicate Tx(s)
Given txHash, attempts to find out duplicate tx(s) present in pending pool.
Tx is considered to be duplicate, when it has, same sender address & nonce
Method : POST
URL : /v1/graphql
query {
pendingDuplicates(hash: "0x2d17f2941e33afd3a648e3257857ed032191b7b93911364ba4906d640ca69b49") {
from
to
gas
gasPrice
hash
nonce
pendingFor
queuedFor
pool
}
}
New pending tx(s)
Listening for any new tx, being added to pending pool, in real-time, over websocket transport
Transport : WebSocket
URL : /v1/graphql
subscription {
newPendingTx{
from
to
gas
gasPrice
nonce
}
}
New confirmed tx(s)
Listening for any new tx, leaving pending pool i.e. confirmed, in real-time, over websocket transport
Transport : WebSocket
URL : /v1/graphql
subscription {
newConfirmedTx{
from
to
gasPrice
}
}
Pending Pool Changes
Whenever any change in pending tx pool happens i.e. tx joins/ leaves pool, subscriber will be notified of those
- Tx joins pending pool, when it's ready to be included in next block [ though might not ]
- Tx leaves pool, when tx it has been included in just mined block
Transport : WebSocket
URL : /v1/graphql
subscription {
pendingPool{
from
to
gasPrice
pool
}
}
New pending tx(s) from
When ever any tx is detected to be entering pending pool, where from address is matching with specified one, subscriber will be notified of it.
Transport : WebSocket
URL : /v1/graphql
subscription {
newPendingTxFrom(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
}
}
New confirmed tx(s) from
When ever any tx is detected to be leaving pending pool i.e. got included in some block, where from address is matching with specified one, subscriber will be notified of it.
Transport : WebSocket
URL : /v1/graphql
subscription {
newConfirmedTxFrom(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
}
}
Catching new pending tx from A
When new tx, sent from address A, is detected to be entering pending pool, client to be notified. Also when tx will be confirmed, they will be notified, that tx has left pending pool.
Transport : WebSocket
URL : /v1/graphql
subscription {
newTxFromAInPendingPool(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
pendingFor
queuedFor
}
}
New pending tx(s) to
When ever any tx is detected to be entering pending pool, where to address is matching with specified one, subscriber will be notified of it.
Note: Tx(s) attempting to deploy contract, will have no
toaddress
Transport : WebSocket
URL : /v1/graphql
subscription {
newPendingTxTo(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
}
}
New confirmed tx(s) to
When ever any tx is detected to be leaving pending pool i.e. got included in some block, where to address is matching with specified one, subscriber will be notified of it.
Transport : WebSocket
URL : /v1/graphql
subscription {
newConfirmedTxTo(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
}
}
Catching new pending tx to A
When new tx, where recipient address is A, is detected to be entering pending pool, client to be notified. Also when tx will be confirmed, they will be notified, that tx has left pending pool.
Transport : WebSocket
URL : /v1/graphql
subscription {
newTxToAInPendingPool(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
pendingFor
queuedFor
}
}
Queued Pool
Queued tx pool inspection APIs.
Queued for more than X
For listing all tx(s) queued for more than or equals to x time unit, send graphQL query
Method : POST
URL : /v1/graphql
query {
queuedForMoreThan(x: "1h10m39s") {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Queued for less than X
For listing all tx(s) queued for less than or equals to x time unit, send graphQL query
Method : POST
URL : /v1/graphql
query {
queuedForLessThan(x: "1m10s100ms") {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Queued from A
For getting a list of all queued tx(s) from specific address, send a graphQL query like
Note : These are present in queued pool due to nonce gap in sender's address i.e. there must be some tx with lower nonce present in pending pool & until that one gets mined, these tx(s) in queued pool, will not move into pending pool.
Method : POST
URL : /v1/graphql
query {
queuedFrom(addr: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313") {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Queued to A
For getting a list of all queued tx(s) sent to specific address, you can send a graphQL query like
Method : POST
URL : /v1/graphql
query {
queuedTo(addr: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313") {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Top X pending
Top X queued transaction(s), with high gas price
Method : POST
URL : /v1/graphql
query {
topXQueuedWithHighGasPrice(x: 10) {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Top X queued transaction(s), with low gas price
Method : POST
URL : /v1/graphql
query {
topXQueuedWithLowGasPrice(x: 10) {
from
gas
gasPrice
hash
input
nonce
to
value
v
r
s
pendingFor
queuedFor
pool
}
}
Queued Duplicate Tx(s)
Given txHash, attempts to find out duplicate tx(s) present in queued pool.
Tx is considered to be duplicate, when it has, same sender address & nonce
Method : POST
URL : /v1/graphql
query {
queuedDuplicates(hash: "0x2d17f2941e33afd3a648e3257857ed032191b7b93911364ba4906d640ca69b49") {
from
to
gas
gasPrice
hash
nonce
pendingFor
queuedFor
pool
}
}
New queued tx(s)
Listening for any new tx, being added to queued pool, in real-time, over websocket transport
Transport : WebSocket
URL : /v1/graphql
subscription {
newQueuedTx{
from
to
gas
gasPrice
nonce
}
}
New unstuck tx(s)
Listening for any new tx, leaving queued tx pool i.e. unstuck, in real-time, over websocket transport
Transport : WebSocket
URL : /v1/graphql
subscription {
newUnstuckTx{
from
to
gasPrice
}
}
Queued Pool Changes
Whenever any change in queued tx pool happens i.e. tx joins/ leaves pool, subscriber will be notified of those
- Tx joins queued pool, due to some issue in sender's account [ mostly nonce gap ], because it's not eligible for inclusion in next block to be mined
- Tx leaves pool, when lower nonce has been filled up & this stuck tx is now ready to get included in block [ It's unstuck now ]
Transport : WebSocket
URL : /v1/graphql
subscription {
queuedPool{
from
to
gasPrice
pool
}
}
New queued tx(s) from
When ever any tx is detected to be entering queued pool ( because they're stuck due to nonce gap ), where from address is matching with specified one, subscriber will be notified of it.
Transport : WebSocket
URL : /v1/graphql
subscription {
newQueuedTxFrom(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
}
}
New unstuck tx(s) from
When ever any tx is detected to be leaving queued pool ( because they were stuck due to nonce gap ), where from address is matching with specified one, subscriber will be notified of it.
Transport : WebSocket
URL : /v1/graphql
subscription {
newUnstuckTxFrom(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
}
}
Catching new queued tx from A
When new tx, from address A, is detected to be entering queued pool, client to be notified. Also when that tx will be unstuck, client will be notified, that tx has left queued pool & it's now eligible to enter pending pool & become candidate tx for next block to be mined.
Transport : WebSocket
URL : /v1/graphql
subscription {
newTxFromAInQueuedPool(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
pendingFor
queuedFor
}
}
New queued tx(s) to
When ever any tx is detected to be entering queued pool ( because they're stuck due to nonce gap ), where to address is matching with specified one, subscriber will be notified of it.
Transport : WebSocket
URL : /v1/graphql
subscription {
newQueuedTxTo(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
}
}
New unstuck tx(s) to
When ever any tx is detected to be leaving queued pool ( because they were stuck due to nonce gap ), where to address is matching with specified one, subscriber will be notified of it.
Transport : WebSocket
URL : /v1/graphql
subscription {
newUnstuckTxTo(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
}
}
Catching new queued tx to A
When new tx, targeted to address A, is detected to be entering queued pool, client to be notified. Also when tx will be unstuck, they will be notified, that tx has left queued pool & it's now eligible to enter pending pool & become candidate tx for next block to be mined.
Transport : WebSocket
URL : /v1/graphql
subscription {
newTxToAInQueuedPool(address: "0x63ec5767F54F6943750A70eB6117EA2D9Ca77313"){
from
to
gasPrice
pendingFor
queuedFor
}
}
GraphQL Playground
harmony packs one graphQL playground for you, where you can play around with both query & subscription methods.
queryworks over HTTP transport, where assubscriptionworks only over Websocket transport.
URI : https://<base-url>/v1/graphql-playground
GraphQL Query/ Subscription Examples
I've written some examples for programmatically querying GraphQL API over HTTP & subscribing to topics for listening to MemPool state changes in real-time, over Websocket transport.
You can find those here. Before you run those
- Make sure you've Python3 installed. They're tested to be working on
Python 3.9.2 - Let's first enable virtual environment, by doing
cd examples
python3 -m venv venv
source venv/bin/activate
- We can now fetch dependencies, by doing
pip install -r requirements.txt
- You can now run any of examples, by doing
python3 query.py
python3 subscribe_1.py
Make sure, you've access to
harmonynode.
- Finally when you're done, you can get out of virtual environment
deactivate
Note:
harmonyis not recommended for use in production environment at time of writing this. It's under active development.
.
128 Jan 6, 2023
24 Dec 26, 2022
8.8k Jan 7, 2023
211 Nov 5, 2022
3.4k Jan 1, 2023
4 May 26, 2022
7 Jun 11, 2022
0 Jul 7, 2022
349 Dec 16, 2022
0 Feb 11, 2022
3.1k Jan 1, 2023
412 Nov 18, 2022
59 Nov 17, 2022
103 Nov 27, 2022
43 Dec 25, 2022