sisong
Applying patch is slow on HDDs
Hi, thank you for making this project.
When we test this, it is fast on an SSD but slow on an HDD when applying patches. The patches can be small but it copies the entire old directory structure, changes the file then deletes the old one. This is slow as opposed to modifying the existing files.
So, could it be fixed to be faster by modifying the existing files? If there is a way to already do this then please let me know.
Additionally, could I pay for this to get done? Will gladly do so
Thanks!
When hpatchz executes dir-patch, the first step is to create the newly added and modified files;
The second step is to copy all the unchanged old files to the new directory if the old directory and the new target directory are different; and if the old and new directories are the same, just perform the merge process without copying.
hpatchz does not currently support full inplace-patch: for example, a large file make small modified in-place to get the target new file.
The only algorithm in this repository that supports inplace-patch is create_inplaceB_lite_diff() & hpatchi_inplaceB(), but this is an overly simplified implementation and is not universal, see demo HPatchLite.
This general algorithm for in-place updates is complex to implement, and it becomes more complex with directory support.
It is recommended to split large projects into multiple data-independent parts and manage them separately, and create patches for each.
That's unfortunate. So you are not willing to be compensated for implementing this feature?
It would already be an improvement to identify only modified files and copy those instead of creating a clone of the directory structure.
If this cannot be done, I will have to look at how to improve because it is a bottleneck.
When the old and new directories are the same, this is already the case.
It would already be an improvement to identify only modified files and copy those instead of creating a clone of the directory structure.
The problem is with things like 9GB of data where there are many large files, some of which might be changed, which is my use case, game files.
While it is true we can work around this to some extent, it would be awesome of this project could gain this feature, beyond HPatchLite.
I have just completed the development of the "diff by CreateSignature" feature,
The next focus of development work may be on compatibility with the .zsync format,
This choice is related to the frequency of users of the demand and the possibility of implementation; for example, after implementation, it may be used in the Linux package manager or replace zsync.
However, general & universal inplace-patch are still relatively difficult. If you can provide multiple versions of the game data for testing, I may include this requirement in the next R&D plan.
I'll reach out to you via email soon with the details of our game and our needs.
There is a problem with this. The in-place update algorithm is more about solving the problem of insufficient disk space, and it comes at the cost of larger patches and a large increase in I/O reads and writes, which is likely to make the patch speed slower!
I/O can only be saved when most of the data does not need to be moved.
It seems that this is not the result you expected.