[Feature Request] Add Option for User-Agent

[sommer-gei]

Hey Adam, thank you for your nice little script! :-)

I ran into the problem that the website I'd like to fetch all URLs from blocked the default wget User-Agent (currently "Wget/1.20.3 (linux-gnu)"). To progress with my task I manually changed your (current) script and added a User-Agent string (to the wget command) and it worked very well.

Question: Are you willing to add an option for the User-Agent?

If yes, I would prepare a PR … :-)

KR

[adamdehaven]

Sure, feel free to submit a PR

[sommer-gei]

FYI: In the meantime I (temp.) changed the User-Agent with a local ~/.wgetrc file with this content

user-agent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0"

[adamdehaven]

@sommer-gei did you ever prepare the Pull Request? I'm happy to add as an option if you want to submit the changes; just please limit the changes to only adding the User Agent option

[sommer-gei]

@adamdehaven Nope, ATM no PR. :-/

(Changing the wget UA manually in the script code is no problem. Implementing/adding the UA as a script CLI parameter is more complex as I thought … #sry)