Add support for Otterscan JSON-RPC API extensions

[ZePedroResende]

trafficstars

Describe the feature

Otterscan is an Ethereum block explorer designed to be run locally with an archive node companion. It's more private compared to other solutions because it allows you to query your node, avoiding sharing sensitive data with third parties. This is achieved without an external indexer through custom RPC methods that currently are only available in Erigon. The addition of this extension (even if behind a feature flag) would allow the community to run Otterscan with a different client other than Erigon. Users would have access to useful information not available on the standard RPC endpoints, without an external indexer, which can be very helpful for data-intensive applications.

### Endpoint checklist
- [x] `ots_getApiLevel`
- [x] `ots_getInternalOperations`
- [x] `ots_hasCode`
- [x] `ots_getTransactionError`
- [x] `ots_traceTransaction`
- [x] `ots_getBlockDetails`
- [x] `ots_getBlockDetailsByHash`
- [x] `ots_getBlockTransactions`
- [ ] `ots_searchTransactionsBefore`
- [ ] `ots_searchTransactionsAfter`
- [ ] `ots_getTransactionBySenderAndNonce`
- [ ] https://github.com/paradigmxyz/reth/issues/8638

Additional context

No response

[naps62]

@gakonst as discussed, can we get this assigned to @ZePedroResende ? I'll be working on this with him

I was thinking a good first step would be to get an initial endpoint done as a draft, mainly to start discussing architecture. I imagine some input will be required, so I want to optimize everyone's time :)

[gakonst]

SGTM! Let us know how to proceed here.

[mattsse]

cool, some pointers: the way it's structure is that we define a trait per namespace with all the functions, like:

https://github.com/paradigmxyz/reth/blob/94ba83f6353cc8d8e25c6acc118b8f6e90145511/crates/rpc/rpc-api/src/web3.rs#L4-L16

and then add a type that implements this like:

https://github.com/paradigmxyz/reth/blob/94ba83f6353cc8d8e25c6acc118b8f6e90145511/crates/rpc/rpc/src/web3.rs#L23-L38

[naps62]

@mattsse thanks, that helped a lot already

what's your preference regarding how to manage the work here? a single large branch where we built the feature and merge as a whole? merging small incremental changes, disabled through a feature flag?

[mattsse]

incremental would be ideal, for example

1. trait bindings
2. impl (can be done in multiple steps)
3. integration in rpc-builder+cli

[naps62]

Bringing back @gakonst 's comment from the PR, feels more on-topic here:

My main Q/concern here is: What new tables are needed to support these APIs? Could we scope them very clearly, as well as understand 1) what stages (if any) we need to add to generate these new tables, 2) how much DB overhead they incur? My main Q/concern here is: What new tables are needed to support these APIs?

makes sense. We went this route because of mattsse's suggestion, but I do admit we still need to learn more about that. also because we're still learning the codebase here, so any input is much appreciated

I was getting more familiar with reth storage because of this topic yesterday. Here's my thoughts so far (I'll let @ZePedroResende correct/complete me):

• ots_getTransactionBySenderAndNonce seems to need a (Address, u64) | H256 table
• ots_getContractCreator should need an Address | Address mapping
• ots_searchTransactionsBefore/After should be by far the biggest one: it requires mapping each address to any tx that ever touched it (outbound, inbound, internal).
• ots_getBlockDetails shouldn't need extra tables, but it does have an issuance object. in Ethereum this is now meaningless, but depending on what other chains are need to support, may need some computation
• ots_getTransactionError and ots_traceTransaction seem like they reuse the existing tracing code?

Maybe some of these can leverage existing tables that I overlooked? I also assume we'll at the very least want to feature-gate these features to avoid the extra overhead unless explicitly needed?

[gakonst]

I also assume we'll at the very least want to feature-gate these features to avoid the extra overhead unless explicitly needed?

100% - for sure.

cc @joshieDo as these are all indices and it might overlap with the future etl work

[ZePedroResende]

Endpoint checklist

• [ ] ots_getApiLevel
• [ ] ots_getInternalOperations
• [ ] ots_hasCode
• [ ] ots_getTransactionError
• [ ] ots_traceTransaction
• [ ] ots_getBlockDetails
• [ ] ots_getBlockDetailsByHash
• [ ] ots_getBlockTransactions
• [ ] ots_searchTransactionsBefore
• [ ] ots_searchTransactionsAfter
• [ ] ots_getTransactionBySenderAndNonce
• [ ] ots_getContractCreator

[onbjerg]

@ZePedroResende moved the tasklist to the main issue

[0xAlcibiades]

This still seems to be missing a few parts including the erigon namespace for otterscan support to work. I see this was recently added to foundry and wonder if there'd be interest to bring things up to speed on this side @Rjected @Evalir @ZePedroResende

[0xAlcibiades]

Relates to: https://github.com/foundry-rs/foundry/pull/5414

[wakamex]

without ots_getContractCreator I can't use the dope new Ape feature which stores contract creation metadata: https://github.com/ApeWorX/ape/pull/2001

[wtdcode]

Anyone working on the ots_searchTransactionsBefore, ots_searchTransactionsAfter and ots_getTransactionBySenderAndNonce? I'm going to give it a try.

[Rjected]

Anyone working on the ots_searchTransactionsBefore, ots_searchTransactionsAfter and ots_getTransactionBySenderAndNonce? I'm going to give it a try.

hey @wtdcode ! w.r.t. new ots endpoints, we want to prioritize those that do not require adding extra indexing to reth, for example ots_getContractCreator looks like it doesn't necessarily need an extra index. I don't know much about ots_searchTransactionsBefore / ots_searchTransactionsAfter / ots_getTransactionBySenderAndNonce, so the first step there would be taking a look at other implementations of those endpoints, and checking if they can be done without extra indices on the reth end. If they do not, we can then help guide an implementation much more easily. Does that make sense?

[0xAlcibiades]

@Rjected has there been any further thinking on this? As a reth operator, it would be very nice to be able to setup otterscan as a viewer for the state of the chain, and for people operating rollups with reth would probably be a big deal not to pay to have a copy of etherscan deployed. Could these methods perhaps be added in an exex?

[jenpaff]

Can this be closed in favour of #13499 ?

[mattsse]

yeah I think we can