ArchiveBot

ArchiveBot copied to clipboard

and should produce the same result as .

JustAnotherArchivist

Display final WARC size (and total downloaded amount) in "has finished" message

1

``` yipdw: perhaps make the bot also give the final total filesize in the "has finished" message? so that you can tell whether it was successful ```

hannahwhy

Move logging/settings code to a separate process

1

The ArchiveBot logging and job settings code has caused jobs to stall and prematurely abort a few times now. It should be moved out of the main wpull process and...

hannahwhy

Build ignore set for Lithium forums

3

PressStar on #archivebot gave us these ignore patterns: ``` tag.*\.js (user|board|blog)-id.*\.js (\w+)-p.*\.js (enable|disable)autocomplete (notifymoderator|reply|login|registration)page searchform(v1|\.form\.form)* kudosbuttonv2 nospellcheck _change_me_ authentication/contributions/actions lightboxrendercomponent renderinlineeditform surveylauncher:getsurveyurl facebookconnectbuttonsecondary ``` I've asked for path prefixes where...

hannahwhy

youtube-dl jobs don't accurately report downloaded size

1

This could be an issue in wpull and/or in the way ArchiveBot uses the wpull size hooks.

hannahwhy

Trying to archive a twitch VOD when I encountered the above error, dashboard output as follows: https://gist.githubusercontent.com/infrequent/ebdc3e5881928c14114a/raw/85c77eb5a41be0493c7def6526ed6662693b5728/gistfile1.txt issued command on IRC: !ao http://www.twitch.tv/shaboozey/v/12288654 --youtube-dl job id: 7p1afhyxcdn0zx79oo8er5uti

infrequent

Automatically pick the fastest pipeline (lowest ping time) for new jobs

1

If a new job is created and there are multiple open pipelines, ping them all and give the job to the one with the lowest ping time. (Props to MrRadar...

Asparagirl

Validate ignore regexes before committing them to jobs

1

``` [14:04:28] the !ig command should probably validate the regex before adding it [14:04:34] DFJustin: good point [14:04:43] it should, but the bot's in Ruby and the pipeline's in python...

hannahwhy

ArchiveBot needs a new command, something like... `!whatareyouworkingon ` or `!pipelinestatus ` ...that gives back some basic pipeline stats, number of jobs it can potentially handle, the number of jobs...

Asparagirl

It would be nice to have a way to see the contents of ignoresets without having to look them up in the source code.

JesseWeinstein