David Foster

jemalloc: Investigate use to reduce memory fragmentation & growth

1

[jemalloc] seems promising for mitigating [long-term growth of memory by Crystal when it has been running for a long time]. [Procedure for running a Python program with jemalloc] (UNVERIFIED): *...

Ability to search for URLs that did not download at all or did not download successfully (with an error)

3

It is common when I'm attempting to do a _full_ download of an entire domain's content to want to verify that I've actually downloaded everything, both at all and without...

Support dynamic rewriting of URLs constructed in JavaScript 🧠

1

Priority: Medium * Impedes archival of a site David cares about: https://bongo.cat/ * ⚠️ Fixing this issue **and** making JS-based dynamic rewriting the default (which we may not want to...

Sometimes it is desirable to delete the downloaded version of a resource. Especially if the resource had an error downloading originally and the user desires to try the download again.

Infinitely scrolling pages: Support downloading a paginated endpoint

1

If you view the page: * https://newsletter.pragmaticengineer.com/archive you will notice that the page's JavaScript makes an initial request to a paginated API endpoint to get the first 12 posts: *...

Some websites - especially image hosting sites - have been observed to refuse download requests with HTTP 403 Forbidden if the HTTP Referer header in the download request doesn't match...

When Crystal is run from source the `--shell` CLI argument can be used to summon an interactive shell that allows the user to do advanced operations against the currently open...

Priority: Low * These days @davidfstr mostly runs incantations in the shell from [CATHODE > workspace.py], which already include imports of model classes. So the absense of these auto-imports is...

Currently the server that Crystal uses when the "View" button is pressed is single-threaded and only supports responding to one connection at a time. Alter it to **support multiple connections...

Recording Links: Sometimes 1-2s delay when creating Resources (fg_task_2)

6

Next/last step: * [ ] [Move database operations off the UI thread to a **dedicated database thread**](https://github.com/davidfstr/Crystal-Web-Archiver/issues/28#issuecomment-1598010879) * Bonus: [Eliminate the "beachball" that happens when opening large projects](https://github.com/davidfstr/Crystal-Web-Archiver/issues/28#issuecomment-1613947664) --- Priority:...