Crystal-Web-Archiver
Crystal-Web-Archiver copied to clipboard
Downloads websites for long-term archival.
[jemalloc] seems promising for mitigating [long-term growth of memory by Crystal when it has been running for a long time]. [Procedure for running a Python program with jemalloc] (UNVERIFIED): *...
It is common when I'm attempting to do a _full_ download of an entire domain's content to want to verify that I've actually downloaded everything, both at all and without...
Priority: Medium * Impedes archival of a site David cares about: https://bongo.cat/ * ⚠️ Fixing this issue **and** making JS-based dynamic rewriting the default (which we may not want to...
Sometimes it is desirable to delete the downloaded version of a resource. Especially if the resource had an error downloading originally and the user desires to try the download again.
If you view the page: * https://newsletter.pragmaticengineer.com/archive you will notice that the page's JavaScript makes an initial request to a paginated API endpoint to get the first 12 posts: *...
Some websites - especially image hosting sites - have been observed to refuse download requests with HTTP 403 Forbidden if the HTTP Referer header in the download request doesn't match...
When Crystal is run from source the `--shell` CLI argument can be used to summon an interactive shell that allows the user to do advanced operations against the currently open...
Priority: Low * These days @davidfstr mostly runs incantations in the shell from [CATHODE > workspace.py], which already include imports of model classes. So the absense of these auto-imports is...
Currently the server that Crystal uses when the "View" button is pressed is single-threaded and only supports responding to one connection at a time. Alter it to **support multiple connections...
Next/last step: * [ ] [Move database operations off the UI thread to a **dedicated database thread**](https://github.com/davidfstr/Crystal-Web-Archiver/issues/28#issuecomment-1598010879) * Bonus: [Eliminate the "beachball" that happens when opening large projects](https://github.com/davidfstr/Crystal-Web-Archiver/issues/28#issuecomment-1613947664) --- Priority:...