Chunk size patch

[Ryabina-io]

Closes #2330. I added +2 to chunk size calculation denominator, as @tilacog advises and our subgraph continued indexing. I don't know internals of table.columns.len() - does it counts full-text search columns for example. But with this PR I want to push progress on this issue somehow. Also I created this PR to 0.26.0 release branch because I tested only that change over 0.26.0 version and I don't know right process.

[lutter]

table.columns does include fulltext columns (see here) but we never insert the vid, that is determined by the database.

Are we sure that that issue is caused by the InsertQuery? Looking at Layout.update, it runs two queries: ClampRangeQuery and InsertQuery. ClampRangeQuery here always uses 2 bind variables so is not the reason for the error. InsertQuery here uses at most table.columns.len() + 1 bind variables per entity, which makes me think the math there is correct.

Since this does fail in practice, there's some mistake in that calculation, and I'd really like to know what that is. Is there any chance you can run a failing subgraph locally and have graph-node print details of the calculation here, in particular, number of entities, number of entity keys, number of columns, and chunk size?

[Ryabina-io]

@lutter added logs in https://github.com/Ryabina-io/graph-node/blob/chunk_size_debug/store/postgres/src/relational.rs#L807 Logs around crashed subgraph:

Jun 23 14:46:47.355 INFO Applying 13789 entity operation(s), block_hash: 0xb9db6d9c41efce3b52d0515e0ed7b1d9d14e6378e8514241b51f44680df5eef4, block_number: 13907188, sgd: 1, subgraph_id: QmchHuvNLcLRGt4C3pLArRXtkECPPCaZqoycKvzpYagsNU, component: SubgraphInstanceManager
Inserting entities entity_type: DelegatorRewardHistoryEntity, number_of_entities: 4683,  number_of_columns: 7, chunk_size: 8191
Inserting entities entity_type: DelegationPoolHistoryEntity, number_of_entities: 2,  number_of_columns: 7, chunk_size: 8191
Inserting entities entity_type: RewardCutHistoryEntity, number_of_entities: 2,  number_of_columns: 9, chunk_size: 6553
Updating entities: GraphAccount, number_of_entities: 2, number_of_entity_keys: 2,  number_of_columns: 23, chunk_size: 2730
Updating entities: DelegatedStake, number_of_entities: 4683, number_of_entity_keys: 4683,  number_of_columns: 18, chunk_size: 3449
Updating entities: Indexer, number_of_entities: 2, number_of_entity_keys: 2,  number_of_columns: 43, chunk_size: 1489
Updating entities: Allocation, number_of_entities: 2, number_of_entity_keys: 2,  number_of_columns: 37, chunk_size: 1724
Updating entities: Poi$, number_of_entities: 1, number_of_entity_keys: 1,  number_of_columns: 2, chunk_size: 21845
Updating entities: SubgraphDeployment, number_of_entities: 2, number_of_entity_keys: 2,  number_of_columns: 28, chunk_size: 2259
Updating entities: Epoch, number_of_entities: 1, number_of_entity_keys: 1,  number_of_columns: 13, chunk_size: 4681
Updating entities: Delegator, number_of_entities: 4405, number_of_entity_keys: 4405,  number_of_columns: 14, chunk_size: 4369
Jun 23 14:46:54.474 ERRO Subgraph failed with non-deterministic error: Error while processing block stream for a subgraph: store error: number of parameters must be between 0 and 65535	, retry_delay_s: 1800, attempt: 213, sgd: 1, subgraph_id: QmchHuvNLcLRGt4C3pLArRXtkECPPCaZqoycKvzpYagsNU, component: SubgraphInstanceManager

[tilacog]

It seems that impl QueryFragment for InsertQuery is counting the bindings correctly: one for each field plus one for the block range.

But InsertQuery delegates walk_ast to its columns via QueryValue, and I'm suspicious that this for loop in the ColumnType::TSVector might be producing extra bindings that we are not accounting for when calculating the chunk size:

https://github.com/graphprotocol/graph-node/blob/28580bb76178bde5e4174dd5f73b1aed0e706b87/store/postgres/src/relational_queries.rs#L596-L616

A single iteration would produce 2 bindings.

I don't know if @Ryabina-io's subgraph even uses that field type, but it might be worth giving a shot at including those bindings in the chunk size estimate.

[lutter]

It seems that impl QueryFragment for InsertQuery is counting the bindings correctly: one for each field plus one for the block range.

But InsertQuery delegates walk_ast to its columns via QueryValue, and I'm suspicious that this for loop in the ColumnType::TSVector might be producing extra bindings that we are not accounting for when calculating the chunk size:

...

A single iteration would produce 2 bindings.

I don't know if @Ryabina-io's subgraph even uses that field type, but it might be worth giving a shot at including those bindings in the chunk size estimate.

Nice find. Yes, that would definitely do it since it will use 2 bindings for each term in the TSVector. That would affect only subgraphs that use fulltext search.

[github-actions[bot]]

This pull request hasn't had any activity for the last 90 days. If there's no more activity over the course of the next 14 days, it will automatically be closed.