ArchiveBot
ArchiveBot copied to clipboard
Ability to add --no-offsite-links mid grab
Would it be possible to add this mid grab (so say if the grab is running out of control/getting spam sites etc)
5vqqwfgciq4y7ksb6e4olk5ao is a key example of when this could be done.
This is controlled by a wpull command-line switch (see https://github.com/ArchiveTeam/ArchiveBot/blob/master/pipeline/archivebot/seesaw/wpull.py#L83-L86).
As far as I know, this is not under wpull script control, so modifications to wpull will be necessary. Paging @chfoo.