Update subdomain sources

[edoardottt]

• [x] #107
• [ ] #108
• [x] #109
• [x] #110
• [ ] #111
• [x] #112
• [x] #113

[ExitoLab]

@edoardottt how will this work? Can you give examples

[edoardottt]

Sure @ExitoLab .

Scilla with the subdomain subcommand tries to retrieve subdomains for a specific inputted domain: e.g.

scilla subdomain -target google.com

The previous command performes a lot of requests against google.com subdomains.

If instead this command is used:

scilla subdomain -target google.com -db -no-check

The flag -db means that we're using API from some online sources in order to retrieve previously collected subdomains.
As example https://crt.sh/?q=%25.google.com&output=json.

At time of writing (12 oct 2023) scilla supports these online sources: anubis, bufferoverrun, crtsh, hackertarget, sonar, threatcrowd, threatminer and virustotal.

The problems are that some of these services stopped working and they are very few.

You can see here in the amass repo https://github.com/owasp-amass/amass/tree/5f1f7176bae60975c1e5be64273cb201f1bb37c3/resources/scripts/api or in the subfinder one https://github.com/projectdiscovery/subfinder/tree/main/v2/pkg/subscraping/sources how many of them are used.

The goal is to remove the not working ones and add new ones to the https://github.com/edoardottt/scilla/tree/main/pkg/opendb package in scilla. You can see in that directory how it works.

I'm gonna create a list of issues specific for each new online service to be added.

[iamshreeram]

@edoardottt , As of today (10/15/23) Below aren't working - bufferoverrun, Sonar Omnisint, threat Crowd

@ExitoLab , We can probably split the work on implementing subdomain sources - subdomaincenter, ahref, dnsrepo, grep.app;

[ExitoLab]

@iamshreeram sure, we can split the work. Which one do you want to work on

[iamshreeram]

@ExitoLab , Im currently working on subdomaincenter. You can pick the next.

@edoardottt - Regarding grep.app as subdomain source - From the Ada codebase, The API (https://grep.app/api/search?regexp=true&q=([_a-zA-Z0-9]{1}[_a-zA-Z0-9-]{0,61}[a-zA-Z0-9]{1}.google.com)) is failing with Forbidden Error.

But, this (https://grep.app/api/search?q=google.com&regexp=true) is returning a payload that is unrelated to the subdomain; Please have a look into this. thanks!

[edoardottt]

Got it...So this https://github.com/owasp-amass/amass/blob/5f1f7176bae60975c1e5be64273cb201f1bb37c3/resources/scripts/api/grepapp.ads#L24 is not working..

We can skip that for now :(