Handle invalid domains more gracefully
๐ฃ Description
As per #42, invalid domains should be handled gracefully.
These are checked for in main() before running the executor, and put in a separate "invalid domains" bucket.
Also, as part of code cleanup, the domain class was moved to a separate python file.
There is a LOT to do and a lot of redundant call chains to be removed, but each journey starts with a single step!
๐ญ Motivation and context
๐งช Testing
Added unit tests for valid/invalid cdnCheck() and main() calls with valid/invalid domains
Sample output
โฏ findcdn list "*.domain.com"
0 Domains Validated.
1 domain(s) rejected, listed below:
*.domain.com
0it [00:00, ?it/s]
{
"date": "11/21/2022, 11:44:09",
"cdn_count": "0",
"domains": {},
"invalid_domains": [
"*.domain.com"
]
}
Domain processing completed.
0 domains had CDN's out of 1.
1 Domains Validated.
2 domain(s) rejected, listed below:
*.foo.bar
wildcard domain
[Pending: 0 jobs]==[Threads: 1]: 100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 1/1 [00:01<00:00, 1.26s/it]
{
"date": "11/21/2022, 11:44:53",
"cdn_count": "1",
"domains": {
"cisa.gov": {
"IP": "'104.117.51.217'",
"cdns": "'.edgekey.net', '.akamaitechnologies.fr'",
"cdns_by_names": "'Akamai', 'Akamai'"
}
},
"invalid_domains": [
"*.foo.bar",
"wildcard domain"
]
}
Domain processing completed.
1 domains had CDN's out of 3.
โ Pre-approval checklist
- [ ] This PR has an informative and human-readable title.
- [ ] Changes are limited to a single goal - eschew scope creep!
- [ ] All future TODOs are captured in issues, which are referenced in code comments.
- [ ] All relevant type-of-change labels have been added.
- [ ] I have read the CONTRIBUTING document.
- [ ] These code changes follow cisagov code standards.
- [ ] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
- [ ] Tests have been added and/or modified to cover the changes in this PR.
- [ ] All new and existing tests pass.
โ Pre-merge checklist
- [ ] Revert dependencies to default branches.
- [ ] Finalize version.
โ Post-merge checklist
- [ ] Add a tag or create a release.
This pull request introduces 1 alert when merging b3b320d1d889fcb73b5187ee4877250eb5d70e43 into 7bbf50e4a1cf1f32999c1abdcabc064d8ebf53f8 - view on LGTM.com
new alerts:
- 1 for Unused import
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down โป completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out our post on the GitHub blog.
I think those "test_list_broken/est_file_broken" tests that fail, were failing before. I actually don't even understand the tests to be honest.
I'll double check with the other branches.
Good point. The tests should probably get a good review as well while I am on this.