Modular Indexer

[ptrwtts]

The Indexer does many jobs, and not everyone needs all of them. We want to make the design more modular, so you can run the Indexer in different modes, to only do what you need. Here are the main things I see:

Order Relaying

It's possible that someone, like a market maker or a marketplace, wants to run their own Indexer purely for submitting orders. Part of the reason is that there is a cost to submit orders (AR tokens), so hosted Indexers will likely need to rate limit it. So power users must do it themselves.

An indexer in this mode would not be able to verify if orders are valid or not. So you would only do this with your own indexer. That said, we probably want to support a feature where it can index all orders from Arweave, and make sure it doesn't submit any duplicates.

Order Indexing

This involves listening to Arweave for relevant transactions, filtering invalid messages, normalizing different order types, and storing them in the orders table for easy access. It only verifies that the order conforms to the spec, NOT whether it is fillable. For many integrators, this is all they need. They have their own systems for maintaining an order book, and just want the raw orders. Sure, they could go to Arweave directly, but this simplifies it for them.

Objective Order Book (Liquidity)

Take the raw orders, and make sense of them. This requires indexing tokens, approvals, etc, to figure out fillability, and includes aggregations like calculating the best price per token to optimize queries. It purely deals with objective, on-chain data (orders, contracts, tokens, sets). Similar to above, for many use cases, they have their own metadata system, and just need liquidity data. E.g. the popular tokens/floor API does is pure liquidity data no metadata (assuming it's queried by contract).

Subjective Order Book (Metadata)

Indexing metadata in order to make the liquidity data more useful. This is the last step to have an end-to-end system for developers who don't already have their own stack. It's also the hardest to do at scale for all collections, which is why there is value in making it optional to go all the way and do this component for all contracts.