Discrepancy in behavior between gRPC API and REST API
🐞 Bug Report
The REST API on a local observer node (using the official image as well as one built from sources) does not provide the same block information available through the gRPC API.
What is the severity of this bug?
I think it's somewhere between these two. It doesn't have to block shipping, but it does mean that the advice in the developer guides to use the REST API is actually bad advice until this is fixed.
Important: * We have to do this before we ship, but it can wait until the next sprint (product or feature won't function without it, but it's not blocking us or users right now). Team should do this in the next sprint.
Should have: * It would be better if we do this before we ship, but it's OK if we don't (product functions without this, but it's a better user experience). Consider adding to a future sprint.
Reproduction steps
Steps to reproduce the behaviour:
I have a container running the official imaged, invoked like this:

I copy-pasted this command from the guide online and modified the paths to match my local filesystem. The log output when this container runs:

I have this configuration for the CLI:

When I run the CLI to get the latest block (via gRPC, per the config), this is the result:

You can see here that the "latest" block synced to the local observer is no more than 9 blocks behind the official testnet access node (blocks seem to come so fast that no doubt a handful flew by while I was entering the command).
However, when I try to do the equivalent thing via the rest interface, I get a very different result:

From what I've read about recent sporks and how blocks propagate, it looks like the rest interface doesn't report any blocks after what looks to me like the root snapshot for devnet spork 37.
Just to validate further, I tried requesting the blocks at height 80989859, which is what was reported as latest by the CLI just a few moments before. This is the result:

As I indicated in slack, I don't observe this behavior when querying access.devnet.nodes.onflow.org.
Expected behaviour
I would expect that I get the exact same information via REST or via gRPC.
Specifications
- official observer node docker image