Enhance logging

[cyb3rjerry]

Hey folks! So I'm currently reviewing the logging within Amass to make it a little more explicit and verbose. My understanding is that we're currently logging via the "color" variables and that sending the data gathered in amass.log is not suitable since there's too much information/noise generated there.

My gut feeling is currently that the logging is a little iffy/not verbose enough. I'd like to propose the following:

1. Add a more debug info in various stages to avoid the user thinking amass is "hanging". For example, at the beggining of the enum command, there's a period where nothing is outputted even though it's actually doing initializaton stuff (see https://discord.com/channels/433729817918308352/433729817918308354/1143382446864338975)
2. Add debugging levels. I feel like this would be very helpful and is something common in various tools such as nmap for example. I personally really like the -v, -vv, -vvv approach as it's an easy way to set debug levels.
3. Start using the log package and wrapping it in "colors" because.. everyone likes colors. The log package is included in the standard lib so this would still follow nicely with the "avoid externals deps when possible" ideology.

Any thoughts on this? I'll be more than glad to stick to the current code styles and ideologies 🙂

[rynmrtn]

thanks for your interest in helping make amass more usable for the average user!

I don't think the colors would be needed for the logs, especially given that its not being written to a rich-text file.

There are a couple options I think worth exploring:

1. Simple: there is a package that hashicorp open sourced that works w/ the std. this would allow the CLI config (or config file) set a log level and automatically ~filter what logs get written to the file based upon the desired verbosity.
2. Not use any sort of external package and determine whether to log based upon the configuration passed through to different parts of the code.
3. Upgrading the version of Go used by amass to 1.21, which includes the more robust slog package. slog would enable level-based logging and open doors for other nice features.

#3 is the most effort; #2 seems potentially messy but would reduce external deps #1 is likely to do the trick but requires an external dependency

Thoughts on pushing up a prototype of #1 and #2 to see how they feel?

[cyb3rjerry]

Sounds like a plan to me :) Will make a POC for one the of smaller sub-commands and will reach back out.

[cyb3rjerry]

@rynmrtn just to make sure we're on the same line here, we're talking about both file AND stdout logging right? I just realized you seem to only be talking about log file. My overall concern is also about the larger stdout debug info. Amass isn't super verbose realtime to it's user and this was the main pain point I was currently facing.

[mzpqnxow]

Hello @cyb3rjerry

I won't put much here but if you have the time, please check out #735

This gist of that is that it'd be nice if stdout/stderr logging could be set to be more friendly to programmatic consumption

I do use the JSON formatted output for the results ("output") but it would be great if all stdout/stderr output could also be emitted as NDJSON

Maybe I'm the only one interested in this :)

[cyb3rjerry]

Hello @cyb3rjerry

I won't put much here but if you have the time, please check out #735

This gist of that is that it'd be nice if stdout/stderr logging could be set to be more friendly to programmatic consumption

I do use the JSON formatted output for the results ("output") but it would be great if all stdout/stderr output could also be emitted as NDJSON

Maybe I'm the only one interested in this :)

Yeah I think this would be reasonable. A lot of tools such as Trufflehog and Snyk for example offer JSON output via the --json flag. This would also enable the tool to be more "compatible" in larger scale environments where output monitoring is important.

Shouldn't be too crazy to do IMO.