wayback_archiver
wayback_archiver copied to clipboard
buren
Ruby gem to send URLs to Wayback Machine
Updates the requirements on [robots](https://github.com/fizx/robots) to permit the latest version. Changelog Sourced from robots's changelog. 0.10.0 Make sure the fetch robots.txt operation happens with a user agent (via rb2k) 0.9.0...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
Updates the requirements on [spidr](https://github.com/postmodern/spidr) to permit the latest version. Changelog Sourced from spidr's changelog. 0.7.0 / 2022-12-31 Added {Spidr.domain} and {Spidr::Agent.domain}. Added {Spidr::Page#gif?}. Added {Spidr::Page#jpeg?}. Added {Spidr::Page#icon?} and {Spidr::Page#ico?}....
The Internet Archive has started to more aggressively rate limit requests and after just a dozen or so requests (with the default concurrency setting `5`). After some testing we even...
wayback machine seems to have lowered the rate again. always was hitting 429 after 1 minute. resolution was to limit requests to 5 per minute works for me, no more...
- [ ] Optimize for Ruby 3.0, [2.7 is EOL March 31, 2023](https://endoflife.date/ruby) so should still be supported in v2 (in my opinion) - [ ] #32. - [ ]...
I got this traceback: ```/usr/lib/ruby/2.7.0/net/http.rb:960:in `initialize': execution expired (Net::OpenTimeout)``` This was with the wayback_archive 1.4.0 gem, reproducible with latest github code. I'm not sure if this project is still maintained,...
Add the ability to write a CSV report to a file.
In `Request#perform_request` ``` 429 Too Many Requests - Check HTTP header for date value, if its not too long in the future wait until then, otherwise return 502 Bad Gateway...
> wayback_archiver > Traceback (most recent call last): > 10: from /usr/local/bin/wayback_archiver:23:in `' > 9: from /usr/local/bin/wayback_archiver:23:in `load' > 8: from /var/lib/gems/2.7.0/gems/wayback_archiver-1.4.0/bin/wayback_archiver:4:in `' > 7: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in `require' > 6:...
Updates the requirements on [rake](https://github.com/ruby/rake) to permit the latest version. Changelog Sourced from rake's changelog. === 13.0.6 Additional fix for #389 Pull request #390 by hsbt === 13.0.5 Fixed the...