BrokenPipeError: [Errno 32] Broken pipe while printing to STDOUT from background jobs

[adityahase]

Sentry has seen a few of these errors in the past. But now this is frequently breaking (~1000 / day).

Guess: Somehow the process reading STDOUT from the job process (rq? supervisor?) stops reading the output (or terminates?!).

When this happens

1. Nothing related to the trace-id is shown in the logs (logs/worker.log and logs/worker.error.log).
2. The failed job persists in the RQ job until the failure_ttl expires.

Reference: https://trace.frappe.cloud/organizations/frappe/discover/homepage/?field=title&field=event.type&field=project&field=user.display&field=timestamp&name=All+Events&project=2&query=title%3A%22BrokenPipeError%3A+%5BErrno+32%5D+Broken+pipe%22&sort=timestamp&statsPeriod=24h&yAxis=count%28%29