Intermittently getting write errors in test-engine-live

[mcshicks]

Intermittently we get write error in test engine live stream, usually after 24 hours. To reproduce the problem you just stream a long time. It happens both inside and outside the streaming docker container, but doesn't happen writing locally (i.e. no S3 enabled).

Here is a partial log and screenshot of debugger showing where it happens:

segment @ 0x275d4e0] Opening '/tmp/dash_segment_input_dir/audio_7757.ts' for writing [mpegts @ 0x275cd00] frame size not set Packaging single segment: audio_7755.ts Available segments for DASHing: audio_7756.ts,audio_7757.ts Packaging single segment: audio_7756.ts Processing segments: /tmp/dash_output_dir/audio_7756.ts.mp4 Creating/updating DASH context. Thu Jan 11 2018 14:30:15 GMT+0000 (UTC) Available segments for DASHing: audio_7757.ts,audio_7756.ts [DASH] Generating segments and MPD 116223 seconds too late DASH-ing file: 15.02s segments 15.00s fragments no sidx used Splitting segments at GOP boundaries DASHing file /tmp/dash_output_dir/audio_7756.ts.mp4
[segment @ 0x275d4e0] Opening '/tmp/dash_segment_input_dir/audio_7758.ts' for writing [mpegts @ 0x275cd00] frame size not set Packaging single segment: audio_7756.ts Available segments for DASHing: audio_7757.ts,audio_7758.ts Packaging single segment: audio_7757.ts Processing segments: /tmp/dash_output_dir/audio_7757.ts.mp4 Creating/updating DASH context. Thu Jan 11 2018 14:30:29 GMT+0000 (UTC) Available segments for DASHing: audio_7758.ts,audio_7757.ts [DASH] Generating segments and MPD 116237 seconds too late [DASH] Generating MPD at time 2018-01-11T14:30:30.253Z
[DASH] Current Period Duration: 116114speed= 1x
DASH-ing file: 15.02s segments 15.00s fragments no sidx used Splitting segments at GOP boundaries Results in: /tmp/dash_output_dir/audio_7756.ts.mp4
The packager has processed: /tmp/dash_output_dir/audio_7756.ts.mp4. Last run was 17897ms ago. Average time between runs: 15000.390020629131ms. DASHing file /tmp/dash_output_dir/audio_7757.ts.mp4
[DASH] Generating MPD at time 2018-01-11T14:30:36.849Z
[DASH] Current Period Duration: 116114 mv: cannot stat '/tmp/dash_output_dir/live.mpd.tmp': No such file or directory [DASH] Error moving file /tmp/dash_output_dir/live.mpd.tmp to /tmp/dash_output_dir/live.mpd: I/O Error Error DASHing file: I/O Errorrate=N/A speed= 1x
Error while processing segments: 1 Waiting for the debugger to disconnect... [segment @ 0x275d4e0] Opening '/tmp/dash_segment_input_dir/audio_7759.ts' for writing [mpegts @ 0x275cd00] frame size not set [segment @ 0x275d4e0] Opening '/tmp/dash_segment_input_dir/audio_7760.ts' for writing