[Feature Request] Meta Block Mined and Indexed Timestamp

[ChrisLahaye]

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

The subgraph query API gives access to the following metadata properties:

• block { hash number number_gte }
• deployment
• hasIndexingErrors

What is the expected behavior?

It would really help to have:

1. the timestamp when the block was mined (currently requires an RPC call)
2. the timestamp when the block was indexed (currently requires frequent polling to observe value change ->gives poll interval as error)

Thank you for your consideration,

Chris

[azf20]

hey @ChrisLahaye - can you clarify what you're looking for - you would like block in the subgraph metadata to be extended to:

{ hash number number_gte timestamp indexTimestamp }

Assuming you meant the timestamp when the block was mined

[leoyvens]

We cannot support indexTimestamp as that would be non-deterministic, that sort of information can only be exposed in the indexing status API.

[ChrisLahaye]

Personally I don't need the block timestamp, although I think for many developers it can be useful to not only have the block number and hash, but also the timestamp (indicating when this block was mined). I am however looking for a way to figure out when that block was processed by the indexer.

[azf20]

OK - what is your use case? Are you trying to work out when to fetch new data from the subgraph?

[ChrisLahaye]

OK - what is your use case? Are you trying to work out when to fetch new data from the subgraph?

Not necessarily, although I would be interested in having a discussion about that as well. On Discord I have seen the recommendation to use a 'block watcher' instead of frequent polling. However, a subscription on blocks using an RPC endpoint typically triggers before that block is processed by the indexer.

We would like to have an idea about the latency / processing delay of our subgraph. Essentially have insight, and in case of large delays notify our users, in how long it takes for a transaction to be picked up and processed in the subgraph. Our current method is polling the meta block number, frequently enough so that we can approximate the processing timestamp by the time the meta block number changed compared to the previous poll, and then use the difference of the current (change) time with the block timestamp returned by the RPC endpoint.

[azf20]

OK interesting - in general the difference should be small, what network are you indexing, and are you currently seeing large delays? In any case, I think you can use the block number returned by _meta, the latest blockNumber from your RPC, and a block frequency estimate to track the delay at a point in time (which I think is what you describe in your current method). You would just like to simplify that by calculating the diff (indexingTimestamp - blockTimestamp) directly on data returned by the _meta field? Note that more information is available via the indexing status API, documented here https://thegraph.com/docs/hostedservice/deploy-subgraph-hosted#checking-subgraph-health This includes the latest block for the chain to compare to the latest subgraph bug

[ChrisLahaye]

@azf20 thank you for getting back to me.

OK interesting - in general the difference should be small, what network are you indexing, and are you currently seeing large delays?

We were seeing large delays on matic mainnet and have worked around the issue by moving most RPC calls to our backend where they can be done in parallel instead of sequentially. Our usual delays are around 5~10s with regular peaks to 30s, no idea whether this is common or not.

In any case, I think you can use the block number returned by _meta, the latest blockNumber from your RPC, and a block frequency estimate to track the delay at a point in time (which I think is what you describe in your current method). You would just like to simplify that by calculating the diff (indexingTimestamp - blockTimestamp) directly on data returned by the _meta field?

Yes that is exactly the method we are using (see https://github.com/lemonadesocial/lemonade-metaverse/blob/develop/src/app/services/watchdog.ts). Being able to do (indexingTimestamp - blockTimestamp) would be great as this would not require a high polling frequency to approximate the indexing timestamp.

[schmidsi]

Similar issue here: https://github.com/graphprotocol/graph-node/issues/3060 but without the indeterministic indexingTimestamp