Hypersync source support

[SozinM]

This Pr adds hypersync integration to the rindexer. Initially i made it with enum, but i think that dynamic dispatch fits better here. We need to choose instance during runtime via config.

This is p1 of the changes, would be nice to move streaming logic to another trait/extend current one, to be able to specialize streaming based on the upstream.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
rindexer-documentation	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 1, 2025 2:12am

[SozinM]

Hi @joshstevens19 I'm implementing hypersync support for rindexer and want to ask you opinion on using enum/dyn Trait for provider selection. Now I've implemented a simple enum approach, but it would be quite easy to switch to dyn Trait

[joshstevens19]

Hi @joshstevens19 I'm implementing hypersync support for rindexer and want to ask you opinion on using enum/dyn Trait for provider selection. Now I've implemented a simple enum approach, but it would be quite easy to switch to dyn Trait

Thanks for this i think how you done it is pretty good i was hoping to also extend this to support ExEx on reth later on (but that's for a later thought) but as dyn trait add runtime costs due to dynamic dispatcher vs enum which uses static dispatcher (which is faster) lets pick performance here so what you done IMHO is good - thoughts? if you are in agreement il do a PR review once its out of draft and rebased.

If you can also add to the docs for this once done as well to say we support hypersync endpoints

[SozinM]

Hi @joshstevens19 I think it's ready for review, had little fight with playground autogen :) In this implementation dyn traits doesn't add much overhead (compared with enums as they will need vtable) Example working, alingside with tests. Current implementation doesn't bring much benefit speedwise but would once we specialize streaming. Also other providers could start implement additional traits.

[joshstevens19]

thanks @SozinM i actually made a slight change today to provider to expose block timestamp do you want to rebase to resolve - https://github.com/joshstevens19/rindexer/commit/d1683fbeeaabf4313424f81708193f81908c5f1f ? will take a look at this PR tomorrow good work

[joshstevens19]

Thanks a lot I’m currently on annual leave till 28th but once back we can get this in 💪💪

[SozinM]

@joshstevens19 Sorry, i'm procrastinating to do final touches, but overall it should work now Will push to complete this before you are back :)

[joshstevens19]

hey is this complete @SozinM ?

[SozinM]

Hi @joshstevens19 yes, should be good now

[joshstevens19]

ok going to need to do some manual tests on this over weekend and make sure all good if so can deploy do want to fix some of the documentation call outs etc - nice job @SozinM

[joshstevens19]

So sorry guys will review this tomorrow/Monday and test it out then merge it good fully forgot 🙏

[joshstevens19]

hey when i build this locally i get

error: failed to run custom build command for `hypersync-net-types v0.9.0`

Caused by:
  process didn't exit successfully: `/Users/joshstevens/code/rindexer/target/debug/build/hypersync-net-types-b32cc79a337da369/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at /Users/joshstevens/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hypersync-net-types-0.9.0/build.rs:5:10:
  compiling schema: Error { kind: Failed, extra: "Failed to execute `capnp --version`: No such file or directory (os error 2). Please verify that version 0.5.2 or higher of the capnp executable is installed on your system. See https://capnproto.org/install.html" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
joshstevens@AMWM-G2136D03XG rindexer % 

@SozinM

its trying to run a build.rs of:

fn main() {
    capnpc::CompilerCommand::new()
        .file("hypersync_net_types.capnp")
        .run()
        .expect("compiling schema");
}

we shouldnt have to install capnp locally or on machine to make work imho - how could we fix this?

[joshstevens19]

hey @ozgrakkurt can anyone else take this PR - closing for now - would of loved this to be finished

[SozinM]

Oh, sorry, completely forget to close it myself Problem with capnp that it's impossible to prebuild them and they need to be build on every compilation So no way to remove this capnp req

[ozgrakkurt]

hey @ozgrakkurt can anyone else take this PR - closing for now - would of loved this to be finished

Hey, I'm not actively working on hypersync anymore. Like Misha wrote, it is very difficult to remove capnp compilation step. Maybe can feature-gate it?