Not all status updates are being consumed by API streams

[lrettig]

They're probably being created faster than the service can consume and stream them. We may need to increase the buffer size to accommodate this:

> grpcurl -plaintext -d '' localhost:9092 spacemesh.v1.NodeService/StatusStream
> go-spacemesh --grpc node
...
2020-07-28T11:51:05.529-0400    INFO    00000.defaultLogger     GRPC NodeService.StatusStream
2020-07-28T11:51:11.033-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:21.028-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:31.028-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:41.030-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:51.029-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:53.429-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:53.429-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.429-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:53.429-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.429-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.429-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.429-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.429-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.438-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:53.438-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.438-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:53.438-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.438-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:51:53.438-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.438-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:51:53.438-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:01.029-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:11.032-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:21.034-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:31.035-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:41.033-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:51.030-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:53.075-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:53.075-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:52:53.075-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:52:53.075-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:52:53.076-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:52:53.076-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:52:53.076-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:53.076-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:52:53.076-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening
2020-07-28T11:52:53.076-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:53.076-0400    INFO    00000.defaultLogger     reported status update
2020-07-28T11:52:53.076-0400    INFO    00000.defaultLogger     not reporting status update as no one is listening

[lrettig]

I saw this error from MeshService.LayerStream today too, so it's definitely happening to multiple services - makes sense during initial sync when data is coming in over the channels faster than the API service can consume and stream it

[moshababo]

@lrettig still relevant?

[lrettig]

Not sure, I'm not up to date on the current status of the API. @dshulyak wants to make some changes here, I think.

[lrettig]

See related thread at https://github.com/spacemeshos/go-spacemesh/pull/3263#discussion_r911632470

Unclear whether this is still an issue, given additional buffering that's been added. To investigate.