Piping streams which come from third party servers

[fkocherovsky]

I use addReadStream API to create a ZIP file based on multiple streams. Works as expected when streams are locally created streams. But there is an issue when streams come from third party server and yazl even doesn't detect the issue. Current implementation takes 'first not done entry', pipes it, and on "end" event takes the next not done entry and pipes it ... When there are many entries to pipe, might pass significant time interval when some next entry/stream is being started handled. Significant means in this case that the third party server decides that some stream is to long idle, and as a result the stream is just closed by the server. And as a result Zip isn't created. that such stream is already aborted. It's what happens when we try to create Zip by yazl based on streams from our S3 server. The solution for such situation is new API which doesn't takes streams like addReadStream does, but takes a function which creates stream just before to start piping this specific stream/file. Something like the following:

ZipFile.prototype.addStreamCreator = function(creator, metadataPath, options) {
   var self = this;
   metadataPath = validateMetadataPath(metadataPath, false);
   if (options == null) options = {};
   var entry = new Entry(metadataPath, false, options);
   self.entries.push(entry);
   entry.setFileDataPumpFunction(async function() {
      creator(metadataPath).then((stream) => {
         entry.state = Entry.FILE_DATA_IN_PROGRESS;
         console.log(`Starting to pump ${metadataPath}`);
         pumpFileDataReadStream(self, entry, stream);
         //pumpEntries(self);
      });
   });
 };

BTW, this is already working and tested function. Thanks