Reports as error for all seen_by_* on a 404 of the RDAP loopkup

[00willo]

On the first looup:

$ wget -q -O- http://127.0.0.1:5730/microsoft.com.au
{"alerts":["YOUR-FIRST-CONTACT","RDAP lookup to https://www.rdap.net/domain/microsoft.com.au returned 404"],"category":"ERROR","freq_score":[5.2609,5.1892],"seen_by_isc":"ERROR","seen_by_web":"ERROR","seen_by_you":"ERROR"}

on second lookup:

$ wget -q -O- http://127.0.0.1:5730/microsoft.com.au
{"alerts":["RDAP lookup to https://www.rdap.net/domain/microsoft.com.au returned 404"],"category":"ERROR","freq_score":[5.2609,5.1892],"seen_by_isc":"ERROR","seen_by_web":"ERROR","seen_by_you":"ERROR"}

I would expect at least seen_by_you to have been updated reflect the timestamp of YOUR-FIRST-CONTACT, so maybe something like this could be implemented.

$ wget -q -O- http://127.0.0.1:5730/microsoft.com.au
{"alerts":["RDAP lookup to https://www.rdap.net/domain/microsoft.com.au returned 404"],"category":"ERROR","freq_score":[5.2609,5.1892],"seen_by_isc":"RDAP Lookup not supported for domain","seen_by_web":"RDAP Lookup not supported for domain","seen_by_you":"Sun, 06 Mar 2022 06:00:36 GMT"}

There are several country TLD that have not implemented RDAP at this stage.

[MarkBaggett]

Thanks for the note. It’s hard to know what the right answer is for errors. I don’t want to cache them or treat them as successful lookups so that it will try again next time it sees the domain. That makes sense for resolvable network issues but not so much for these unsupported country code domains.

How would you like to see the errors handles? I think I’m just caching the error for 24 hours to limit the network load but otherwise they are ignored.

Mark

---

From: Graham Williamson @.> Sent: Friday, March 11, 2022 10:41:55 PM To: MarkBaggett/domain_stats @.> Cc: Subscribed @.**> Subject: [MarkBaggett/domain_stats] Reports as error for all seen_by_ on a 404 of the RDAP loopkup (Issue #28)

On the first looup:

$ wget -q -O- http://127.0.0.1:5730/microsoft.com.au {"alerts":["YOUR-FIRST-CONTACT","RDAP lookup to https://www.rdap.net/domain/microsoft.com.au returned 404"],"category":"ERROR","freq_score":[5.2609,5.1892],"seen_by_isc":"ERROR","seen_by_web":"ERROR","seen_by_you":"ERROR"}

on second lookup:

$ wget -q -O- http://127.0.0.1:5730/microsoft.com.au {"alerts":["RDAP lookup to https://www.rdap.net/domain/microsoft.com.au returned 404"],"category":"ERROR","freq_score":[5.2609,5.1892],"seen_by_isc":"ERROR","seen_by_web":"ERROR","seen_by_you":"ERROR"}

I would expect at least seen_by_you to have been updated reflect the timestamp of YOUR-FIRST-CONTACT, so maybe something like this could be implemented.

$ wget -q -O- http://127.0.0.1:5730/microsoft.com.au {"alerts":["RDAP lookup to https://www.rdap.net/domain/microsoft.com.au returned 404"],"category":"ERROR","freq_score":[5.2609,5.1892],"seen_by_isc":"RDAP Lookup not supported for domain","seen_by_web":"RDAP Lookup not supported for domain","seen_by_you":"Sun, 06 Mar 2022 06:00:36 GMT"}

There are several country TLD that have not implemented RDAP at this stage.

— Reply to this email directly, view it on GitHubhttps://github.com/MarkBaggett/domain_stats/issues/28, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFVSDMAICS6BCHROJISHYTU7QHAHANCNFSM5QRJZAXQ. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[00willo]

@MarkBaggett thanks for this handy tool.

I'd be happy with some caching. If the cache time could be a configurable option, I think that would be best. Defenders using the tool, can then make their own decision on how often they'll do failed lookups, and 24 hours seems like a sensible default.

[MarkBaggett]

The cache time is currently configurable. Configure the "rdap_error_ttl_days" entry in the yaml. You can use fractions for less than a day.