Known-sensitive data in API responses should be auto-scrubbed from TF_LOG=trace logs

[skyzyx]

Terraform Version

Terraform v0.12.26
+ provider.newrelic v1.18.0

Affected Resource(s)

The provider itself.

Terraform Configuration Files

Not directly relevant.

Debug Output

https://gist.github.com/skyzyx/a80928a32f6f7f35ddb79ed182973fe9

Expected Behavior

Known-sensitive data that comes back in API responses should be masked/scrubbed/redacted automatically when running Terraform with TF_LOG=trace set. This would greatly simplify the ability to share debug output for these bug reports.

Actual Behavior

I spent nearly an hour going through a 7,000-line trace log, manually scrubbing out email addresses, private integration endpoints, passwords, employee names, etc.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. TF_LOG=trace terraform apply. The execution should complete without error.

[jthurman42]

Thank you for reporting the issue.

Digging through the gist, seems that most of the exposed TRACE level data is from logging the request body/headers, does that sound accurate?

I've created an upstream ticket to address the HTTP request logging at newrelic/newrelic-client-go#327

[skyzyx]

It does sound accurate. Thank you for taking a look.

[skyzyx]

I don't have a proper script for this — just search+replace snippets for my IDE, but maybe this will help to kick-start efforts.

First block is search, second block is replace.

---

First and last names with email inside <>

\\"(name|recipients)\\":\\"(\w[A-Za-z\*]+,?\s?){2,3}\s+<[^@]+@[^>]+>\\"

\"$1\":\"First Last <*****@*****.com>\"

Empty name with email inside <>

\\"(name|recipients)\\":\\"\s*<[^@][email protected]>\\"

\"$1\":\"First Last <*****@*****.com>\"

Only email address, without <>

\\"(name|recipients)\\":\\"[^@][email protected]\\"

\"$1\":\"First Last <*****@*****.com>\"

Specific first, middle initial, and last name with email inside <>

Not a regex as much as a string replacement.

Bob T Jones <[email protected]>

First Last <*****@*****.com>

URL pointing to another vendor

https:\/\/customer([^\.]*)\.vendor\.com\/api\/now\/table\/em_event

https://*****.com/api/event

Username and password

\{\\"username\\":\\"([^\\]+)\\",\\"password\\":\\"([^\\]+)\\"\}

{\"username\":\"REDACTED\",\"password\":\"REDACTED\"}

New Relic Account ID

https:\/\/connect\.newrelic\.com\/(\d+)

https://connect.newrelic.com/0000000

Admin API Key

NRAA-([0-9a-zA-Z]+)

NRAA-**********

Anonymized Confluence installation

https:\/\/confluence\.domain\.com\/pages\/viewpage\.action\?pageId=(\d+)

https://confluence/pages/viewpage.action?pageId=0000000

Anonymized Slack web hooks

https:\/\/hooks\.slack\.com/([^/]+)/([^/]+)/([^/]+)/([0-9A-Za-z]+)

https://hooks.slack.com/*******

[kidk]

Thanks for the feature request. At this time we won't make the investment to scrub all sensitive data from the TF_LOG=trace logs. It will be a game of whack a mole, and we don't feel it's the right path forward. As an alternative we will improve our instructions to encrypt the data you sent to us, so it's only visible to New Relic employees. I'm also working with our security team to find a way to easily share temporary data with us.

[tarciosaraiva]

FWIW using TF_LOG=debug exposes the API key sent on requests - might expose other information as well. At a minimum this should be scrubbed or not logged.