Is it possible to throttle the concurrency? or how would this be added?

[jfrux]

Any clues?

[indexzero]

It's tricky because we visit all the URLs in a graph fashion without keeping track of currently open connections. You'd have to track that state and then it would be possible.