joshstevens19
Support Reth nodes to index using ExExes
Reth now support execution extensions we should support ability to config an Reth node and stream data using the ExExes instead of over JSONRPC
https://www.paradigm.xyz/2024/05/reth-exex
For additional context, a major benefit of ExEx is re-org awareness meaning you can live index rather than maintaining a probabilistic safe distance
There is also now native support for backfilling, see https://github.com/paradigmxyz/reth-exex-examples/tree/main/backfill
interested to work on this! @joshstevens19 this should be straightforward, do you mind if I take a crack at this?
Hey @SkandaBhat would love someone to take this so yeah please do it 🙏 shouldn’t be to complex main thing here is:
- as this has to happen on startup of reth we should design it so the CLI makes it easy to run the commands as it seems you basically have to run reth under hood so for example rindexer start should also start reth up to set the ExEx connection
- any config for reth should live in the rindexer yaml making it feel like one
- we should reuse everything rindexer does handle wise but just skipping the fetching and using reth to fetch but the rest should hit the main logic
- we should make sure we backport the data as well and not just index the latest stuff
shout if you need any help look forward to the PR
I also going to test and merge this in the morning just FYI https://github.com/joshstevens19/rindexer/pull/103
Looking into this today, should have a draft early next week @joshstevens19
@joshstevens19 sorry about this - ended up getting caught up with something else and haven't been able to wrap this up. I might be able to look into this early next week, but if someone else is willing to take this up, please go ahead and assign them.
sure @SkandaBhat think the PR would be super great so yeah feel free to pick it up next week
@joshstevens19 Quick question here - do we want to run reth in-process?
We could probably keep rindexer lightweight and instead use exex remotely - https://github.com/paradigmxyz/reth-exex-examples/blob/1f74410740ac996276a84ee72003f4f9cf041491/remote/bin/exex.rs