wayback_archiver
wayback_archiver copied to clipboard
Update spidr requirement from ~> 0.6.1 to >= 0.6.1, < 0.8.0
Updates the requirements on spidr to permit the latest version.
Changelog
Sourced from spidr's changelog.
0.7.0 / 2022-12-31
- Added {Spidr.domain} and {Spidr::Agent.domain}.
- Added {Spidr::Page#gif?}.
- Added {Spidr::Page#jpeg?}.
- Added {Spidr::Page#icon?} and {Spidr::Page#ico?}.
- Added {Spidr::Page#png?}.
- {Spidr.proxy=} and {Spidr::Agent#proxy=} can now accept a
String
or aURI::HTTP
object.0.6.1 / 2019-10-24
- Check for the opaque component of URIs before attempting to set the path component (
@kyaroch
). This fixesURI::InvalidURIError: path conflicts with opaque
exceptions.- Fix
@robots
instance variable warning (@spk
).0.6.0 / 2016-08-04
- Added {Spidr::Proxy}.
- Added more options to {Spidr::Agent#initialize}:
:default_headers
: specifies the default headers to set in all requests (@maccman
).:limit
: specify the maximum number of links to visit.:open_timeout
,:read_timeout
,:ssl_timeout
,:continue_timeout
, and:keep_alive_timeout
: setsNet::HTTP
timeouts.- Allow {Spidr::Settings::Proxy#proxy= Spidr.proxy=} to accept
nil
.- Use
Net::HTTPResponse#get_fields
in {Spidr::Page} to correctly return multiple values for repeated headers.- Fixed a bug in {Spidr::Page#method_missing} where method names were not being correctly converted to header names.
- Fixed a bug in {Spidr::Page#cookie_params} where
Set-Cookie
flags were not being filtered out.- Rewrote the specs to use webmock and increased spec coverage.
0.5.0 / 2016-01-03
Added support for respecting
robots.txt
files.Spidr.site('http://reddit.com/', robots: true)
Added {Spidr.robots=} and {Spidr.robots?}.
Added {Spidr::Page#each_mailto} and {Spidr::Page#mailtos}.
Fixed a bug in {Spidr::Agent.host} that limited spidering to only
http://
URIs.Rescue
Zlib::Error
to catchZlib::DataError
andZlib::BufError
exceptions caused by web servers that use incompatible gzip compression.Fixed a bug in {URI.expand_path} where
/../foo
was being expanded tofoo
instead of/foo
.
... (truncated)
Commits
65d8c87
Addyard-spellcheck
.90ee108
Documentation fixes.42dac9f
Updated the copyright years.b6c6b28
Ensure all top-level method examples are passed a block.f8169c9
Add an example for usingSpidr.domain
.ac2cbfe
Merge branch '0.7.0'd946128
Updated the ChangeLog for 0.7.0.da008b5
Added an.editorconfig
file.118bfd1
Fixed an exception message.0e88254
Re-ordered@param
arguments.- Additional commits viewable in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
-
@dependabot rebase
will rebase this PR -
@dependabot recreate
will recreate this PR, overwriting any edits that have been made to it -
@dependabot merge
will merge this PR after your CI passes on it -
@dependabot squash and merge
will squash and merge this PR after your CI passes on it -
@dependabot cancel merge
will cancel a previously requested merge and block automerging -
@dependabot reopen
will reopen this PR if it is closed -
@dependabot close
will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually -
@dependabot ignore this major version
will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) -
@dependabot ignore this minor version
will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) -
@dependabot ignore this dependency
will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)