Callback from file writing process

[sheikn]

Is your feature request related to a problem? Please describe. This library is great. However, when compressing lots of files (or large files) where the resulting archive is several hundred megabytes, there is a noticeable delay between compression progress reaching 100% (as calculated from the ProgressCallback) and when the BitCompressor.Compress method returns. My guess is that this is when the archive file is actually written to disk?

Describe the solution you'd like Would it be possible to get another callback routine that describes this delay? e.g void setWriteCallback( const ProgressCallback &callback )

[rikyoz]

Hi!

This library is great. However, when compressing lots of files (or large files) where the resulting archive is several hundred megabytes, there is a noticeable delay between compression progress reaching 100% (as calculated from the ProgressCallback) and when the BitCompressor.Compress method returns.

Umm, interesting! I have noticed 7-zip File Manager behaving similarly, however at 99%, not 100%, and not every time. Probably is not bit7z's fault, since it relies on the public callback interfaces exposed by the 7-zip DLLs, and these latter are the ones that ultimately call bit7z's callbacks.

My guess is that this is when the archive file is actually written to disk?

Yes, it could be an explanation for this behaviour. However, I suggest also to perform the following checks:

• How do you calculate the progress percentage? Rounding instead of truncating, for instance, can give a different percentage and hence a 99% becomes a 100%, while 7-zip is still compressing/writing the archive. In general, you could check the exact values (in bytes) of the arguments of the progress and total callback and see if they indeed become equal with a significant delay from the compress function returning time.
• Does the official 7-zip GUI app behave the same when compressing the same files with the same format? If it is so, it is a "bug" of 7-zip itself, which I cannot solve.
• Which version of 7-zip are you using? Maybe most recent releases have solved the bug or reduced the delay.
• Are you updating an existing archive? Usually, bit7z returns almost immediately after 7-zip completes the compression. However, a delay may happen if you had updated an archive: at the end of the operation, bit7z must delete the old file before returning, and this possibly introduces a delay in certain conditions.

Would it be possible to get another callback routine that describes this delay? e.g void setWriteCallback( const ProgressCallback &callback )

Unfortunately, 7-zip does not provide such callback (as far as I know), so bit7z has no control over the writing process and the relative progress, and it cannot offer such callback to the user.

[sheikn]

The 100% delay was down to rounding. The actual delay, as you have pointed out is at 99%. I have done some more tests using the 7-zip file manager (version 19.00 32bit version, build date 21/02/2019). I can't see any delay at 99% however. In all cases, I'm creating new archives. The overall time taken from start to completion is the same using this library, and the 7Zip filemanager, so it looks like 7Zip filemanager takes the file writing process into consideration for its progress dialogs.

[rikyoz]

The 100% delay was down to rounding. The actual delay, as you have pointed out is at 99%. I have done some more tests using the 7-zip file manager (version 19.00 32bit version, build date 21/02/2019). I can't see any delay at 99% however. In all cases, I'm creating new archives. The overall time taken from start to completion is the same using this library, and the 7Zip filemanager, so it looks like 7Zip filemanager takes the file writing process into consideration for its progress dialogs.

I did some testing, and I managed to replicate your issue only in one case:

• 7z archive format
• Solid compression
• Compressing big Windows x86 executable files (e.g., setup files in the order hundreds of megabytes)¹
• Multi-threading²

In this particular scenario, the progress rapidly goes up to 99% and then stops for a while before completing. However... with these exact settings also 7-zip has the same issue, hence I cannot do anything in bit7z to solve it: it is the DLL that is calling bit7z's callbacks at the wrong time. Please, could you confirm that this is also your case? And if not, could you give me more details about the kind of files you are compressing, as well as other settings you use for the operation? Thanks in advance!

¹ I suspect that the reason is the 7-zip's BCJ filter, given that it is applied only to x86 executables. ² Even though bit7z v3.x does not specify the number of threads to be used by 7-zip, probably this latter automatically selects it. I've tested by forcing to use a single thread, and this issue does not appear, in both 7-zip GUI and bit7z. Hence, it is probably an issue related to the BCJ filter used in multi-threading.

[sheikn]

Sorry about the long delay in my reply. I'm only sporadically working on this project at the moment. We are using this in a backup process, and the files being backed up can be very large; sometimes a couple of gigabytes in size. The archive format is always 7z in this case as it results in considerable file size savings over the zip format. I'm not explicitly specifying solid archives, so maybe its defaulting to that? And I'm also not explicitly specifying multiple threads.

Is there any way to specify the number of threads used by 7-zip?