steam-lancache-prefill icon indicating copy to clipboard operation
steam-lancache-prefill copied to clipboard

Published 20 hours ago • verified publisher icon tpill90

Download only as needed

Open zontreck opened this issue 1 year ago • 9 comments

This PR adds a small change that makes prefill do a HEAD request, and check if X-Cache-Status is set to HIT. If it is, then it can safely skip and move onto the next chunk. Avoids unnecessary downloads from the cache server.

zontreck avatar Nov 08 '23 00:11 zontreck

This is useful, for instance, when a download is interrupted. As some modern games can be as large as 100+ GB, and while steam has many, many chunks and URLs to check, this still speeds up the process of resuming the download, or skipping games that the cache already has.

zontreck avatar Nov 09 '23 02:11 zontreck

Thanks for the pr. I'll probably be able to take a look at this in the next few days or so. Will let you know any feedback then

tpill90 avatar Nov 09 '23 02:11 tpill90

After some updates have released, i have confirmed my patch holds up to those as well, only downloading the changed stuff, and skipping by the rest that the cache already has. Hope this makes it in.

zontreck avatar Nov 29 '23 10:11 zontreck

I've finally been table to take some time to check this out, and test it out a bit. It does work and certainly speeds up things by a lot! However I have two reservations about including these changes:

  1. This change breaks the benchmark command's functionality. Since clients would no longer be pulling the actual download over LAN, then running a benchmark as a client will produce incorrect results. Running this as a client on my desktop achieved speeds of over 450gbit/s which would never be possible by a single client. Keeping the benchmark functionality is very important as it is depended on by the community to identify bottlenecks in their setups.

  2. I am also a bit cautious to change the download behavior to be different than what the Steam client itself does. SteamPrefill has always been intended to match the real client's behavior as much as possible, so that there can be the confidence that downloading via Steam/SteamPrefill will have the exact same result.

tpill90 avatar Dec 01 '23 00:12 tpill90

Hmm perhaps put this behind a command line toggle then to not break the default behavior of other commands?

zontreck avatar Dec 01 '23 00:12 zontreck

I can modify this to put it behind a command line toggle, or just fix it so that benchmark and other such commands will not utilize the HEAD request check.

zontreck avatar Dec 01 '23 00:12 zontreck

Are you on discord? Would you maybe be interested in joining the lancache discord and chatting about this a bit there?

tpill90 avatar Dec 01 '23 01:12 tpill90

Yea i'm on discord.

zontreck avatar Dec 01 '23 01:12 zontreck

I updated the files in the PR. I added a command line toggle to turn on skipping chunks that have already been downloaded. Updated it to be compatible with upstream/master. By default, this will use the current behavior of not skipping chunks.

I wasn't sure where the best place for this toggle would've been. Nor was i sure what the best nickname for the flag was.

zontreck avatar Mar 24 '24 08:03 zontreck