spidr
spidr copied to clipboard
Store history queue in Hash of host:port and paths.
To reduce lookup time in the Spidr::Agent#queue
, we can store the URLs in a Hash of the unique host:port
pair and the URL paths. This will also facilitate events for when new hosts are seen.