Provide a control for subgraph pruning

[lutter]

With #3666, operators can control pruning of individual subgraphs, but it requires user action for each subgraph that should be pruned.

There should be a control that automatically prunes subgraphs without the need to explicitly run graphman prune. There are a few options for this:

• Make pruning part of the subgraph manifest so that subgraph authors can indicate how much history should be retained
• Allow operators to set how much history to retain through configuration, e.g., for each chain in graph-node.toml, or through additional rules in graph-node.toml based on subgraph names

Making pruning part of the manifest has the downside that subgraphs that only differ in how much history they retain will have different IPFS hashes, even though they are really the same subgraph, and can all be used to serve the same queries, assuming that the query is for a block height within the subgraph's available history. As an example, a subgraph that retains all history and a subgraph that only retains the last 1000 blocks of history can both be used to respond to a query at a block that is less than 1000 blocks behind the subgraph head.

[azf20]

That is indeed a drawback of the Manifest Hash as unique identifier. Presumably to change that pattern would be a very significant refactor? We have previously discussed identifying discrete and re-usable indexed regions of the manifest - and perhaps decoupling the indexing definition (datasources + startblocks + mappings + entities) from the subgraph definition would be a step in that direction, but that looks like a rabbithole. It is relevant to other query only settings, which would also include changes to the Schema which don't impact indexing, for example the addition of derived fields.

In terms of indexer controls, I agree that wholesale and subgraph specific settings would probably be useful here.

[github-actions[bot]]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

[azf20]

Having discussed this "in the round", proposing that we allow for control by both subgraph developers and operators.

1. Add "history" to the subgraph manifest:

history: 10000

This is the clearest way for subgraph developers to radiate intent, but it creates limited flexibility and no scope for re-use.

Considerations:

• Should we make this configurable by network? i.e. history: - ethereum: 10000 (This goes against the fact that in practice each subgraph deployment must use the same network across data sources)
• What happens if an indexer wants to override the history for a given deployment? -> indexer configuration should take precedence in this case (and won't impact POI)

2. Let operators set default history per network in graph-node.toml

[chains.mainnet]
history = 10000

They can set individual history by deployment using graphman prune

Proposing the following order of preference when pruning an individual deployment:

1. Deployment configuration
2. Manifest
3. Graph node default

So if the indexer has applied graphman prune with a given block range, that should be applied. If no deployment configuration has been applied, and there is a manifest configuration, that should be applied. Otherwise the default should be applied, if present.

Pruning is a "one way door", and as such appropriate warnings should be provided when altering or overriding a pruning setting, and graceful explanation should be provided when increasing the history_blocks, to indicate that this will not work for existing block ranges, but future indexing will respect the new setting.

[lutter]

2. Let operators set default history per network in `graph-node.toml`

[chains.mainnet]
history = 10000

I am generally ok with what's proposed here, but a per-network setting is a very big hammer, and will probably not be all that suitable in most cases. I feel that a mechanism like the one in #4325 would be better. We could also make pruning something that you can set through [[deployment.rule]] though I feel that is overloading that construct and mixing two different purposes.

[azf20]

Understood - in that case proposing that we focus the scope of this issue to the manifest definition, agree that deployment.rule and chain-level config is maybe not the right place for this.

[incrypto32]

@leoyvens @azf20 Following up on this discussion here

There have been users who wants control on the pruning height (maybe to keep the ability to graft from x blocks behind the chainHead). Right now we do this manually for them when they ping us and ask to prune X blocks.

Why dont we have an optional history on prune and user can choose to provide it?

indexerHints:
 - prune:
     history: 10000

[leoyvens]

I get curious on what benefit this control gives them. How do they know they will need to graft 10k blocks behind the chain head, but not 11k blocks behind chain head?

But I do not oppose to having the history parameter, as long as it's optional, as you mention.

[leoyvens]

There is an optimization we should exploit from the knowledge that a subgraph will be pruned from deployment time. Right now, we create GIST indexes on reference columns (foreign keys): https://github.com/graphprotocol/graph-node/blob/25c720a07939111c42340e1f59320dc75a1eafd2/store/postgres/src/relational/ddl.rs#L216-L222 I propose that pruned subgraphs always use a btree index, by updating the condition to be if immutable || pruned. For a couple of reasons:

1. We know that the subgraph will be pruned, therefore the block_range will not be very selective, so we can assume that a regular btree index will result in at least equivalent performance.
2. We've seen GIST be much larger than their BTree equivalent, so using btree here will be a significant factor in DB space savings.

[azf20]

That sounds good @incrypto32. I can see a case where a developer might want to keep (say) 250k blocks of history, rather than 10k, in case they want to be able to graft onto the subgraph, or if they know that they want monthly rolling calculations, but still want to benefit from getting rid of much older data

Btree suggestion also makes sense, while we are at it.

[incrypto32]

Not sure if this has been discussed, We could just enable ongoing pruning for the subgraph that has history blocks declared in the manifest right? Or do we mean this as just a hint for the indexer to do if they want to?

Currently the admin json RPC accepts the history_blocks here we could just parse the history field from the manifest and then set this history_blocks field and ongoing pruning will take care of the pruning?

[leoyvens]

This is something @lutter can confirm, but afaik all the infrastructure for ongoing pruning is there and we just need to use it just like that flag does.

[lutter]

This is something @lutter can confirm, but afaik all the infrastructure for ongoing pruning is there and we just need to use it just like that flag does.

Yes, all it takes is setting the history_blocks field when the subgraph is deployed - ongoing pruning will then happen automatically according to the GRAPH_STORE_HISTORY_* settings. When it's set at deploy time, there's not even a need to do an initial prune like graphman prune would do since there is nothing to prune.

[leoyvens]

Summarizing discussions in meetings, things I understood us to have agreed about:

• This feature goes in a new spec version.
• The new default, if no hint is specified, is to prune at an operator-defined height.
• A number of blocks of history to keep can be specified.
• It can also be specified to keep all the history.

We are open to alternatives over putting this in the manifest. But as moving forward with this is a priority, we're proceeding with a prototype syntax in the manifest since that is a simple implementation.

Open syntax questions:

1. Do we need an explicit hint syntax to specify "prune to operator-defined height"? IMO it seems fine to not have an explicit form, and have that behaviour only if no hint is set.
2. Bikeshed the hint syntax. Considering that the default is to prune, the hint is actually more about not pruning, what @incrypto32 proposes in the PR makes more sense:

indexingHints:
   - historyBlocks: 100

How do we signal all the history? Maybe historyBlocks: all?

[leoyvens]

On btree vs gist index, @lutter questioned how much history is necessary for the gist index to become advantageous. I don't have a strong opinion on what to do here, as long as we used btree in the pruned default. So while we don't have data to put a precise number on this, I'm fine with using gist even at a relatively short history of say 500 blocks. Or so we don't have to choose a number at all, use btree only for the pruned default and gist if any explicit historyBlocks is set.

[incrypto32]

@lutter thoughts on the above, how do we want to proceed ?

[leoyvens]

The decision was to not pursue changes to the index type right now. We can do that at any time in the future when we have more data to give us confidence in that change.