prefect
prefect copied to clipboard
Race condition writing results and emitting completion events
Discovered while load testing TaskWorkers:
When getting the result of a deferred task, I've seen a few instances of this:
ValueError: Path /task-storage/6d782288-d74b-48ab-ab88-80c13cc255dc does not exist.
or
pydantic_core._pydantic_core.ValidationError: 1 validation error for PersistedResultBlob
Invalid JSON: EOF while parsing a value at line 1 column 0 [type=json_invalid, input_value=b'', input_type=bytes]
For further information visit https://errors.pydantic.dev/2.7/v/json_invalid
What's happening here appears to be that we're emitting the Completed event before writing the results (because we've started to defer writing results until transactions are committed). This happens very slightly after emitting the event, so it's possible that under heavy load, the task engine doesn't write the result until after an outside party sees the future resolve.
Relates to #14098