alloy icon indicating copy to clipboard operation
alloy copied to clipboard

feat(remote/aws): support AWS Secrets Manager as remote component

Open hainenber opened this issue 1 year ago • 3 comments

PR Description

Which issue(s) this PR fixes

Closes #689

Notes to the Reviewer

  • Should I poll the Secrets Manager API continuously to fetch latest secret. Pricing per API calls for Secrets Manager is much expensive compared to AWS SSM Parameter Store and AWS S3 API calls so I'm hesitant to implement poller. Would like to hear everyone thoughts about it.
  • I expose the inner component.Component in componenttest in order to verify the component's health status. Is this A-OK?

PR Checklist

  • [x] CHANGELOG.md updated
  • [x] Documentation added
  • [x] Tests added

hainenber avatar Apr 30 '24 12:04 hainenber

I think it would be nice if it could poll but set the default to not poll. Most of our pulling mechanisms support it so it would be odd if this one did not have that capability.

mattdurham avatar Apr 30 '24 12:04 mattdurham

Thanks Matt! I'll implement the poller and fix whatever makes the tests failed.

Maybe in the long run, we can have a reusable implementation. But that's for another day, I suppose :D

hainenber avatar Apr 30 '24 14:04 hainenber

Docs look ok.

clayton-cornell avatar May 13 '24 16:05 clayton-cornell

I think it would be nice if it could poll but set the default to not poll. Most of our pulling mechanisms support it so it would be odd if this one did not have that capability.

@mattdurham Can you help me understand setting the default to not poll for updates? Don't all of our components default to polling in the background?

rfratto avatar Jul 10 '24 12:07 rfratto

My thought on the default to not polling is related to this comment Pricing per API calls for Secrets Manager is much expensive compared to AWS SSM Parameter Store and AWS S3 API. if it is expensive default to no but allow it to be poll. Also fine with the other direction, default to a low poll rate.

mattdurham avatar Jul 10 '24 13:07 mattdurham

Comparing the pricing of AWS Secrets Manager to S3 does show that secrets manager is more expensive, but perhaps not significantly:

  • AWS Secrets manager: $0.05 per 10,000 API calls
  • S3: $0.004 per 10,000 GET calls

It will take 200,000 polls of secrets for it to cost $1 USD. If we set the polling interval to hourly, and you had 100 Alloy instances all running the same component, that would still only be 73,000 polls per month ($0.36).

It seems like the cost is reasonable enough if we used a 1h polling interval by default. If you're running Alloy in an environment where you need AWS secrets manager (likely Enterprise-like), running Alloy itself probably costs orders of magnitude more money than the secrets manager bill would be.

WDYT @mattdurham?

rfratto avatar Jul 10 '24 14:07 rfratto