Improve multiple endpoints configuration

[stwiname]

Description

Currently specifying endpoints is fairly limited. We can only specify a string for each one. But there are many use cases where you would want to have more control over these endpoints. This could include weighting them, specifying headers or rate limiting options.

Different api providers have different ways of providing api keys and other authentication. Currently the only way to do this is with path params and we convert them to headers. This is not very flexible and can lead to logging leaking api keys.

It is nearly impossible to determine the rate limit for an endpoint, being able to specify this will allow for optimal indexing speeds given a rate limited endpoint.

Requirements

• Be backwards compatible with current flags and project manifests
• Introduce a new data structure to specify endpoints details (this is a rough example):

type EndpointConfig = {
  headers?: Record<string, unknown>;
  rateLimit?: { requests: number, interval: 'second' | 'minute' | 'hour' };
  weighting: number; // TBD, this should allow users to specify a primary endpoint that gets used over other endpoints
  batchSize?: number; // JSON RPC batch size, where applicable (ethereum)
  retries?: number; // Number of retries this endpoint will make before throwing. Default 1.
}

• Allow specifying a default EndpointConfig that applies to all endpoints

Manifests network endpoint would then have the following type string | string[] | Record<string, EndpointConfig>

• CLI flags should accept json strings for this new functionality.
• We should ensure that running a project from a config file works with this feature.