webtorrent
webtorrent copied to clipboard
webtorrent
feat: use storage (FSA+IDB) instead of memory in browser
What is the purpose of this pull request? (put an "X" next to item)
[ ] Documentation update [ ] Bug fix [X] New feature [ ] Other, please explain:
What changes did you make? (Give an overview) Added an option to use drive storage instead of memory in browser.
Replaced memory chunk store with hybrid chunk store which will try to figure out how much space the user has on their drive, use all of it - 16MB, then fallback to memory [should this be blobs instead of memory?] for the rest, this increases the capacity on firefox and safari by 2^31 or 2.14GB, and on chrome by as much as the user has space.
Additionally on most chromium versions the user can specify a rootDir, which allows the user to specify a custom target download folder, which replicates the folder and file structure in the same way as node does.
FSA specific:
Because of the limitations of the FSA API where you cant write and save from the same file at once, as long as the torrent is active it will use x2 as much drive space for each piece downloaded, which means if no pieces are downloaded [if all already exist it won't use any extra], but the moment the tab closes, crashes, minimizes, freezes or turns off in any way that data should be cleared, fairly reliably, and even if somehow they did not get cleared, the very next time the PAGE loads the EXCESS data will be removed.
If the browser closes too fast, and it doesn't manage to remove the store in time [ex: multi-gigabyte torrent], on the next load of the page, the data will be removed.
These last 2 paragraphs have a chance of failing however in very extreme cases, [ex: sudden power cut, full termination of all browser processes] the user needs to be aware of this, there's a chance the data will leak when unwanted to, I did everything possible to minimize the chances of this happening, the only way to make this 100% reliable, is having the user specify if the store will be destroyed or not, when the torrent is first created, which isn't viable for many reasons [page user might not want to do that, or might decide to do that in the moment, non-standard store options and parameters, etc]
Note: starting feb 22, an in-place write might be possible to reduce the doubled storage usage, but we'll see how that pans out see this
Drive store specific:
By default the library still only uses memory, this is because unlike node, the browser doesn't always gracefully close, especially on mobile, I did this to retain legacy behavior, but this needs to be re-considered. If the user wants to reliably delete the data as before, but save the files to drive to allow more capacity, he needs to destroy the torrent with its store AT LASTEST on the beforeunload event, which implies the loss of BFC, or potential loss of functionality/data when the page is suspended by the browser to save RAM when there are a lot of tabs open. IMO this isn't something webtorrent should handle [by for example rescanning store on page unfreeze], but the user should somehow handle beforehand in their app. https://github.com/webtorrent/webtorrent/pull/1903 would be very beneficial for this, because the user could simply re-scan and re-download the missing data after a page wakes up from being frozen/suspended.
Note: this uses ESM, because might as well. Needs to be compiled externally for now xd
Which issue (if any) does this pull request address? https://github.com/orgs/webtorrent/projects/1#card-60702730
Is there anything you'd like reviewers to focus on? Structure, the new package, if the store is trying to do too many things at once, the increasing amt of store opts and how to possibly reduce this?, current defaults.
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?
@DiegoRBaquero I've removed the github dependencies, and pulled latest, you can test the functionality of this over at https://haven.pages.dev/torrent-client/public
If hybrid-chunk-store is ESM, webtorrent can't yet support it
yes, but that can easily be changed, I could compile hybrid to 2 files, one module and one normal, focus on its code pls
🎁 Hey @ThaUnknown, you have just received U$ 1 from @nthypes!
@nthypes thanks for your support! ❤️
@ThaUnknown, you can check your balance at https://givebk.io.
═══
(powered by https://givebk.io) ID: 99cd17f6-ec2b-4234-ae19-662af378dc46
forgot this went to v2 and kinda borked it, fixed merge conflict and updated hcs version
@DiegoRBaquero I've updated the package, also made it pass all these tests: https://github.com/ThaUnknown/hybrid-chunk-store/blob/master/test.js which are probably more strict than any of the tests used by the stores that we currently use, this is as good to merge as it's probably gonna get
:tada: This PR is included in version 2.0.0 :tada:
The release is available on:
Your semantic-release bot :package::rocket:
