Add newPendingRawTransactions to filter system to save bandwidth

[t-anyu]

Hey,

In use cases where client subscribes to newPendingTransactions and gets txhashes would typically re-request transaction content from the node. In public nodes or nodes that serve many simultaneous clients, this would cause some additional load.

It would be nice, and quite easy to implement another subscription of newPendingRawTransactions, which would directly send the raw transaction instead of just tx hash. Then the client can decode the raw transaction without re-requesting json'd eth_getTransactionByHash.

I think this should be easily implemented. Instead of sending tx.Hash(), a tx.MarshalBinary() would be sent to the subscription channel.

(I can do the commit if this gets green light from the maintainers)

[karalabe]

Could you provide some more details about what you're trying to use it for? We do have a few ideas in mind too, but curious about your case.

[t-anyu]

Hey,

In my case, the node that I'm hosting is mostly being used for various trading related applications. But currently working on a MEV tracking case, where we need to record all the pending transactions across multiple nodes.

I believe in most use cases that uses pending transaction filter are being used in tranding / gaming related applications. And in these cases, the exact transaction data is needed.

[amirilidan78]

it would be nice for example i want to listen on new pending transactions and check if receiver address is in my database , execute callback for my application ,but now because i have only txId and i should call 1 http per tx it would not be cool

trezor blockbook is good example it provider transaction detail on new pending transaction not just txId https://github.com/trezor/blockbook/blob/master/docs/api.md subscribeNewTransaction

example

{
    "id": "1",
    "data": {
        "txid": "ca32a8ea2a881ab508e3fa84f03a5ae5073bab977444bcfd5e75d6ce5d31b86f",
        "version": 2,
        "vin": [
            {
                "txid": "75eff4aaabfeaa7824337ef61ad12918ea2a8659918898330949efe9703840bf",
                "vout": 1,
                "sequence": 4294967295,
                "n": 0,
                "addresses": [
                    "bc1qunxnu8gv8ckdsvewqj809ajfyxxfkv6chak2j7"
                ],
                "isAddress": true,
                "value": "129911"
            },
            {
                "txid": "f973e88c40ea8462af2c79432a193c79247d44b24edbe44598e7fceb0361859c",
                "vout": 1,
                "sequence": 4294967295,
                "n": 1,
                "addresses": [
                    "bc1q9zexgk4r956yvskdlpvvz3y9h27e5pa424v4r5"
                ],
                "isAddress": true,
                "value": "98397"
            },
            {
                "txid": "3218619a5f858adf2471ef3eb4263ecdb041cd49f53553ce624f363838f528e2",
                "vout": 1,
                "sequence": 4294967295,
                "n": 2,
                "addresses": [
                    "bc1quc7yudz95x6m7dhh96nfpts4d6e9uxczyh5lx0"
                ],
                "isAddress": true,
                "value": "29939"
            },
            {
                "txid": "55413f9a41dd887762e8ae21917fa95ebed7bfcaabd316ff82035cae441505d8",
                "vout": 1,
                "sequence": 4294967295,
                "n": 3,
                "addresses": [
                    "bc1qww3n386xqfzzrygkq2gjv8jxl47ak8d8frwu2s"
                ],
                "isAddress": true,
                "value": "4229"
            }
        ],
        "vout": [
            {
                "value": "258804",
                "n": 0,
                "hex": "76a914cc75f99318cbeb65673396b53d9298729e558e7688ac",
                "addresses": [
                    "1Ke6BB3hJKzkPdLoYmWmBiuKbeJqpKgp8D"
                ],
                "isAddress": true
            }
        ],
        "blockHeight": 0,
        "confirmations": 0,
        "blockTime": 1654771422,
        "value": "258804",
        "valueIn": "262476",
        "fees": "3672",
        "hex": "02000000000104bf403870e9ef49093398889159862aea1829d11af67e332478aafeabaaf4ef750100000000ffffffff9c856103ebfce79845e4db4eb2447d24793c192a43792caf6284ea408ce873f90100000000ffffffffe228f53838364f62ce5335f549cd41b0cd3e26b43eef7124df8a855f9a6118320100000000ffffffffd8051544ae5c0382ff16d3abcabfd7be5ea97f9121aee8627788dd419a3f41550100000000ffffffff01f4f20300000000001976a914cc75f99318cbeb65673396b53d9298729e558e7688ac02483045022100ae8ddcf6b5a13aa900afb9e5fed523b6db5f7f8a43fcfff8f0904d73aa38fe3f022068ae9717d3512e763f603b0f77fc13da1c0db056f67a217f89c1aee7e80e27370121028e9e92b4719a30cf5934a7692709e63ed956a40b4eea7a93e9ac11efa26ab6d4024830450221009f6596fa13ee6896b6d6e948cf1ee5cdc047e80bcd15ef89e60cb13bb6d0b22002203410f27ac19ec5260eaf5b198fea0f344ea8c5cba8054973020e17f2a75b956201210238d4124e2afa0601cdabc73ef0f9aa0b738ba576f5bac48db8d8307c3dade51a0247304402206bb199089b582a7778eddda7c00def4561c60b617bb53379b29ec3a29d3fa378022077c4a18849e09d0e0f29933e3dacb0944b122a78b61547370e0c1e8d3254132b012102a2c4211dd67e3cff7789f9170f6082b0e80f48c9e16fc9f43bc89a6b1e0713a9024830450221008d077900f6ee0a619e6aebf93827b8df85afdf73eb67811bd924d6cd1bc41d5202205c68c608ba3929d1a12b1d241fc442067c9b011c1590f7e8f1626e67b0b686e5012102d502c2bd743743b71aaef37973f87173d58bcd9e062c581e5d504f12e85b453f00000000"
    }
}

[lmittmann]

@karalabe would you accept PRs that provide the functionality?

[karalabe]

We should review and merge the feature I guess.

[Anti83]

@lmittmann Hi, I merged your PR into my fork, but still getting only hashes. I am using web3 web3.eth.subscribe('pendingTransactions') What is the right way to use it? Thanks.

[t-anyu]

Hey,

I have implemented newPendingRawTransactions on my test node and I suggest that raw RLP is sent instead of Json if performance is concern. RLP encoding seems to be much faster than json encode. Probably RLP is Pooled and now need to ecrecover + keccak?

[ofarukcaki]

Hey,

I have implemented newPendingRawTransactions on my test node and I suggest that raw RLP is sent instead of Json if performance is concern. RLP encoding seems to be much faster than json encode. Probably RLP is Pooled and now need to ecrecover + keccak?

Hey, can you share the implementation details with us? Thanks

[lmittmann]

@lmittmann Hi, I merged your PR into my fork, but still getting only hashes. I am using web3 web3.eth.subscribe('pendingTransactions') What is the right way to use it? Thanks.

@Anti83 You have to update your RPC client to send the additional fullTx flag. See https://github.com/ethereum/go-ethereum/blob/4a69d7607d58685e8f045ff2035d7e8c7f93203e/ethclient/gethclient/gethclient.go#L178-L181 for reference

[ofarukcaki]

@lmittmann Hi, I merged your PR into my fork, but still getting only hashes. I am using web3 web3.eth.subscribe('pendingTransactions') What is the right way to use it? Thanks.

Did you figure out what is the right way?

[holiman]

Fixed by https://github.com/ethereum/go-ethereum/pull/25186

[amirilidan78]

thanks 😍

[jwelch-qn]

Hey all, is it possible to toggle this back to receiving only tx hashes with the latest update? @holiman

[jsvisa]

@jwelch-qn It did have supported, ref https://geth.ethereum.org/docs/interacting-with-geth/rpc/pubsub#newpendingtransactions

eg:

$ wscat -c ws://localhost:8546
Connected (press CTRL+C to quit)

> {"id": 1, "jsonrpc": "2.0", "method": "eth_subscribe", "params": ["newPendingTransactions", false]}
< {"jsonrpc":"2.0","id":1,"result":"0xb94e3d49419b39284360092ec6eec2ea"}

# here comes the new tx with hash only
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0xf73d808cb819267b97dadb98d83a7348f296807c058a526b40cc7594b8f2d66a"}}
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0x7e032b3fa572484f9416a95d4b0a0228c51ccb9f48de9e58c8c24ba38128afd4"}}

[damartripamungkas]

@jwelch-qn It did have supported, ref https://geth.ethereum.org/docs/interacting-with-geth/rpc/pubsub#newpendingtransactions

eg:

$ wscat -c ws://localhost:8546
Connected (press CTRL+C to quit)

> {"id": 1, "jsonrpc": "2.0", "method": "eth_subscribe", "params": ["newPendingTransactions", false]}
< {"jsonrpc":"2.0","id":1,"result":"0xb94e3d49419b39284360092ec6eec2ea"}

# here comes the new tx with hash only
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0xf73d808cb819267b97dadb98d83a7348f296807c058a526b40cc7594b8f2d66a"}}
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0x7e032b3fa572484f9416a95d4b0a0228c51ccb9f48de9e58c8c24ba38128afd4"}}

what is version your geth?

[jwelch-qn]

Hmm, this is helpful, but this change is causing issues with eth_newPendingTransactionFilter and eth_getFilterChanges which is breaking downstream libraries. e.g. https://github.com/web3j/web3j/release

@damartripamungkas @jsvisa

[damartripamungkas]

you must be extend class Web3Subscription, refrence : https://docs.web3js.org/guides/events_subscriptions/custom_subscriptions/

or use my package for subscription : https://github.com/damartripamungkas/simple-client-eth-rpc/tree/master

[jsvisa]

@jwelch-qn It did have supported, ref https://geth.ethereum.org/docs/interacting-with-geth/rpc/pubsub#newpendingtransactions

eg:

$ wscat -c ws://localhost:8546

Connected (press CTRL+C to quit)

{"id": 1, "jsonrpc": "2.0", "method": "eth_subscribe", "params": ["newPendingTransactions", false]}

< {"jsonrpc":"2.0","id":1,"result":"0xb94e3d49419b39284360092ec6eec2ea"}

here comes the new tx with hash only

< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0xf73d808cb819267b97dadb98d83a7348f296807c058a526b40cc7594b8f2d66a"}}

< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0x7e032b3fa572484f9416a95d4b0a0228c51ccb9f48de9e58c8c24ba38128afd4"}}

what is version your geth?

v1.12.0

[jsvisa]

Hmm, this is helpful, but this change is causing issues with eth_newPendingTransactionFilter and eth_getFilterChanges which is breaking downstream libraries.

e.g. https://github.com/web3j/web3j/release

@damartripamungkas @jsvisa

imo this is the web3j’s issue, not geth’s, and you should file an issue in web3j’s repo