crusty icon indicating copy to clipboard operation
crusty copied to clipboard

Investigate hiccups during startup

Open let4be opened this issue 4 years ago • 2 comments

It's especially noticeable when using shorter living jobs

soft_timeout = 30s hard_timeout = 60s

throughput falls almost 3 times...

let4be avatar Jun 23 '21 22:06 let4be

Could be redirect related... Often we would get a redirect to a subdomain or even totally different domain - which we treat as job bounds violation,

=> meaning we've spent some resources and now have to discard this job while inserting a newly discovered domain into the global resolver Q which is usually incredibly congested, so chances are it never makes it(will be discarded as overflow)

With high percentage of jobs throwing redirects leading to job bounds violation chances are we can be stuck in this vicious cycle for a while...

let4be avatar Jun 26 '21 17:06 let4be

Plan is.

  1. Dedicated DNS resolver pools - capacity splited between seeding and domain discovery
  2. Newly started job is a seed and it's first task will be using seeding DNS resolver via .await style calls(i.e. we would delegate resolving to a pool and await result asynchronously)
  3. Seed task is allowed to violate typical same-domain bounds as long as resolved DNS addr_key stays the same(meaning while we violate domain bounds we stay within the same addr_key)
  4. As soon as we finish with seed task every other task would use the domain name seed task ended up using
  5. While seed task is progressing through redirects we should not emit those as "newly discovered domains"
  6. On Job finish we should also record all "aliases" of a checked domain so that if we find any of those we'd know we already checked it

This should eliminate resource waste in Jobs that start with redirects that change source domain name

let4be avatar Jun 27 '21 12:06 let4be