spidr icon indicating copy to clipboard operation
spidr copied to clipboard

Store history queue in Hash of host:port and paths.

Open postmodern opened this issue 13 years ago • 0 comments

To reduce lookup time in the Spidr::Agent#queue, we can store the URLs in a Hash of the unique host:port pair and the URL paths. This will also facilitate events for when new hosts are seen.

postmodern avatar Oct 16 '10 20:10 postmodern