zipstreamer icon indicating copy to clipboard operation
zipstreamer copied to clipboard

Published 20 hours ago • verified publisher icon scosman

Log warnings for unretrievable URLs in descriptor

Open danlamanna opened this issue 2 years ago • 1 comments

These lines silently ignore URLs that can't be retrieved: https://github.com/scosman/zipstreamer/blob/ebcdb4258485f768077a5c7214944cd2e01dfa07/zip_streamer/zip_streamer.go#L37-L44

danlamanna avatar Jan 26 '23 13:01 danlamanna

P2: if anyone want to tackle this would take to master, but not a top issue. Would need to be behind a config flag ZS_LOG_UPSTREAM_ERRORS which still would default off.

Why I don't think this is on by default:

  • Logs could be filled with non-exceptional errors. Ignoring bad URLs is the right thing, some users could send thousands, polluting the log with errors that don't need to be addressed
  • the URLs can contain access keys (examples: S3 access keys as query string param, I used with a system with 256 bit keys in URL). We shouldn't assume the logging platform is setup for "secrets", and getting this should be opt in.

scosman avatar Jan 30 '23 16:01 scosman