Cannot subscribe to new pending transactions

[paymog]

Describe the bug I cannot subscribe to new pending transactions when running a local testnet. However, I can subscribe to new blocks.

To Reproduce Steps to reproduce the behavior:

1. Run the local testnet with docker-compose -f docker-compose.yml -f docker-compose.ts-batch-submitter.yml up
2. Wait for all the services to come alive
3. In another terminal tab run wscat ws://localhost:8546 and in that console issue the following: {"id": 1, "method": "eth_subscribe", "params": ["newPendingTransactions"]}
4. Notice that the subscription is successfully acknowledged
5. Run the following js script with node which will submit a transaction and theoretically subscribe to new pending events: https://gist.github.com/paymog/1dee62da2fbed88d85dba311e8f56142 (note that these pending events come from https://docs.ethers.io/v5/api/providers/provider/#Provider--log-methods)
6. Notice that the event handler is never executed in the wscat terminal nor in the node program

Expected behavior A transaction hash should have been returned in both the wscat terminal and the node program that's subscribed to pending events.

Screenshots Here's the output of the node program when I run it:

/root/.nvm/versions/node/v17.6.0/bin/node /root/.nvm/versions/node/v17.6.0/lib/node_modules/npm/bin/npm-cli.js run run --scripts-prepend-node-path=auto

> [email protected] run
> node index.js

balance is 4999999999999999777944
{
  nonce: 8,
  gasPrice: BigNumber { _hex: '0x01', _isBigNumber: true },
  gasLimit: BigNumber { _hex: '0x1e8480', _isBigNumber: true },
  to: '0x00000398232e2064f896018496b4B44b3d627511',
  value: BigNumber { _hex: '0x64', _isBigNumber: true },
  data: '0x',
  chainId: 1088,
  v: 2212,
  r: '0xb34a1fcee125f977608f12fe89f46c5b6d10115de180d80f9a191f87e36ea0cf',
  s: '0x6df5f8714d493f1e4b6347d27cd8400f806e2b6798d777d4017ca1eab29c662c',
  from: '0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266',
  hash: '0xec40a82b9e3da8e153c618516f2ebb7b8eb8e871aa832ec14abb68ca31f53663',
  type: null,
  confirmations: 0,
  wait: [Function (anonymous)]
}

<note that it hangs here and does not terminate>

System Specs:

• OS: ubuntu 21
• Package Version (or commit hash): 03f1cfda19be4a3f2ca3bfe183c25d41c0edc105

Additional context In the provided gist, if I change "pending" to "block" then the event listener gets correctly triggered.

[paymog]

Looks like this is happening because L2 transactions do not get added to the txPool and instead get applied as a sequencer transaction:

https://github.com/ethereum-optimism/optimism/blob/51a527b8e3fe69940fb8c0f5e4aa2e0ae8ee294c/l2geth/eth/api_backend.go#L299-L316

[paymog]

and these docs seem to confirm: https://community.optimism.io/docs/how-optimism-works/#block-production

Maybe I should be subscribed to new pending transactions on the L1?

[paymog]

Yup, when subscribing on the L1 i get events, as seen in this gist: https://gist.github.com/abdafcc432839975e76027cba0786f71

[smartcontracts]

cc @tynes

[paymog]

Another interesting discovery, if I swap out TxPool() for syncService in the following method, then my L2 transaction subscription works well:

https://github.com/ethereum-optimism/optimism/blob/51a527b8e3fe69940fb8c0f5e4aa2e0ae8ee294c/l2geth/eth/api_backend.go#L352

Ideally we would combine the TxPool() and syncService subscriptions into one (which I'm currently trying to figure out)

[qbzzt]

Another interesting discovery, if I swap out TxPool() for syncService in the following method, then my L2 transaction subscription works well:

https://github.com/ethereum-optimism/optimism/blob/51a527b8e3fe69940fb8c0f5e4aa2e0ae8ee294c/l2geth/eth/api_backend.go#L352

Ideally we would combine the TxPool() and syncService subscriptions into one (which I'm currently trying to figure out)

This makes sense. Currently there is just one sequencer, so there is no need for a transaction pool.

[tynes]

In the existing architecture, there isn't a mempool so subscribing to pending transactions will not work. I see you have looked into the code and found out why. Perhaps you could subscribe to new head instead?

In the future we will be moving to a system that does have a mempool, so the web3 api should work 1:1

[paymog]

Looks like subscribing to the new transactions on the sync service gets what I need. I've made a new filter which returns the entire transaction object as determined by the sync service. One new discovery is that when I try to send the entire transaction object over the wire to a client all the data comes through except for the hash. Here's an example of a subscription using wscat:

 # ❮❮❮ wscat -c ws://localhost:8546
Connected (press CTRL+C to quit)
> {"id": 1, "method": "eth_subscribe", "params": ["newAcceptedTransactions"]}
< {"jsonrpc":"2.0","id":1,"result":"0x2a2c16da483de16a3a093f9107cdc35f"}

< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0x2a2c16da483de16a3a093f9107cdc35f","result":{"nonce":"0x2","gasPrice":"0x1","gas":"0x1e8480","to":"0x00000398232e2064f896018496b4b44b3d627511","value":"0x64","input":"0x","v":"0x8a3","r":"0xd874e1870361e44d936385423a69f89664a66dba96f8fc5aa6098ad1b18950e4","s":"0x478fae4d1933ede3813891e6e58e81558ca7052e5ae35885f8568c6d748a4ccf","hash":null}}}

Note that "hash":null is being returned. I'm trying to figure out why this is and one idea I have is that txDataMarshaling does not have a Hash field: https://github.com/ethereum-optimism/optimism/blob/67146ed3db1a4a1a308afc3b86d5c92beba822e7/l2geth/core/types/transaction.go#L65-L74

Is that it? Might there be some other reason?

[paymog]

Here's a sample PR which demonstrates the issue above (where hash is empty for full transaction objects sent over the wire): https://github.com/paymog/optimism/pull/1

[SZharkov]

@paymog Could you explain please whether you were able to set up pending txs listener and how?