bezzad
[Feature Request] Pre-allocated and stream into final file location.
Would it be possible to have an option to make Downloader pre-allocate the final file and stream into it directly instead of using temporary files? With large downloads the moving of temp files into the final file can take some time, especially with an overzealous anti-virus program.
Implementing this feature while downloading in multiple parts is a really difficult feature. But I try to think about it and if I can solve the problem, I will inform you here. Thanks
@bezzad is it hard?
file = File.OpenWrite(path);
file.SetLength(size);
file.Dispose();
// done.
@GihanSoft It is not enough to just allocate the initial space on the file stream. You have to write to a file at the same time as parallel, which is impossible with today's file system. If you want to do the locking method, the speed will be very slow. I found a solution to this problem but unfortunately, it did not work on the Linux operating system. For this reason, I have postponed doing this feature for later.
@bezzad
I have an idea. what if it run a thread (or task) that copy downloaded data from temp file to final file in a loop?
And only this thread use final file so no concurrency file access problem.
Loop should work parallel to a download with same life time. A TaskCompletionSource can be used to wait for any change or download end.
@GihanSoft thanks for spending your time in this library ;) I feel that you do not realize this issue. We do not want to use the temp file at all. We want to save the download directly to the final file and not have the cost of transferring data from Temp to the final file at the end of the download. Suppose 8 parallel download tasks are completed at the same time. Now your parallel thread starts copying each of the temp files onto the final file. So what is the positive value of this work? We are looking for a method or stream that can fill a file in parallel in different indexes. To do this, please see the following link about MemoryMappedFile: https://github.com/bezzad/FileParallelWrite
Suppose 8 parallel download tasks are completed at the same time. Now your parallel thread starts copying each of the temp files onto the final file. So what is the positive value of this work?
you got it wrong. parallel copy works same time as downloading not at end of downloading. and the "temp files" also can be memory not file.
@GihanSoft Yes, it can be done if a block of data is sent to it in parallel from each task, but this method has its complexities, which in the link I mentioned, this problem is easily solved and should be changed in a good way.
Just looking over this, couldn't we just add a configuration option like "DirectDownload" and change the FileStorage to instead of creating a temp file, create the file that it is downloading, and also disable the StoreDownloadedFile if that option is enabled? The only caveat is that you need to have parallel mode disabled
@Unnoen , @GihanSoft , @CarbonNeuron new version (3.0.0-beta) release and you can see the new feature on this version, now. If the feature you requested was completed in this version, please let us know so that the final version can be published. The changes in this version are fundamental and that is why this version is beta.
You can use the option below to pre-allocate the space required to save the file before starting the download. Even if its value is false, the downloader will still store bytes instead of temps in the destination path. But it will start filling the file with zero bytes size.
// Before starting the download, reserve the storage space of the file as file size, the default value is false
ReserveStorageSpaceBeforeStartingDownload = true;
From now on, the downloader does not use temporary files or RAM to store the data of the chunks and all the bytes are stored directly and concurrently on the final file.
I close this issue because this feature was added in the last version. if you have a problem with that, please create a new issue.