Imperative streaming API does not update the ITaskWithStatus correctly

[phaumer]

The request() method in packages/rest/src/client/AbstractRestClient.ts implements a streaming API for downloading and uploading files. For that the caller can provide a ITaskWithStatus with the request options to monitor the progress. However, if used programmatically in the SDK via a nodejs app that task cannot be used to decide if the file is successfully downloaded and available for reading. This would be a very important requirement for consumers who want to download content such as data sets to a file and then process these files.

The problem is that the task is updated with TaskStage.COMPLETE at the beginning of the onEnd() method before even calling the stream.onEnd(). onEnd() only indicates that all the content is being received. However, that does not mean that at this point all the content has been written and flushed to a file that was used for the stream.

This is only the case when the finish event is called, which only happens after the stream.end(). See the nodejs docs here https://nodejs.org/docs/latest-v12.x/api/stream.html#stream_event_finish. The impact is that code that gets called right after the request finishes and task is changed to complete will still find an incomplete file that does not have all the content.

I propose to implement also the writer.on('finish') handler that then updates the task to complete.