prefect icon indicating copy to clipboard operation
prefect copied to clipboard

Race condition writing results and emitting completion events

Open chrisguidry opened this issue 1 year ago • 1 comments

Discovered while load testing TaskWorkers:

When getting the result of a deferred task, I've seen a few instances of this:

ValueError: Path /task-storage/6d782288-d74b-48ab-ab88-80c13cc255dc does not exist.

or

pydantic_core._pydantic_core.ValidationError: 1 validation error for PersistedResultBlob
  Invalid JSON: EOF while parsing a value at line 1 column 0 [type=json_invalid, input_value=b'', input_type=bytes]
    For further information visit https://errors.pydantic.dev/2.7/v/json_invalid

What's happening here appears to be that we're emitting the Completed event before writing the results (because we've started to defer writing results until transactions are committed). This happens very slightly after emitting the event, so it's possible that under heavy load, the task engine doesn't write the result until after an outside party sees the future resolve.

chrisguidry avatar Jun 20 '24 18:06 chrisguidry

Relates to #14098

chrisguidry avatar Jun 21 '24 14:06 chrisguidry