alloy
alloy copied to clipboard
feat(remote/aws): support AWS Secrets Manager as remote component
PR Description
Which issue(s) this PR fixes
Closes #689
Notes to the Reviewer
- Should I poll the Secrets Manager API continuously to fetch latest secret. Pricing per API calls for Secrets Manager is much expensive compared to AWS SSM Parameter Store and AWS S3 API calls so I'm hesitant to implement poller. Would like to hear everyone thoughts about it.
- I expose the inner
component.Componentincomponenttestin order to verify the component's health status. Is this A-OK?
PR Checklist
- [x] CHANGELOG.md updated
- [x] Documentation added
- [x] Tests added
I think it would be nice if it could poll but set the default to not poll. Most of our pulling mechanisms support it so it would be odd if this one did not have that capability.
Thanks Matt! I'll implement the poller and fix whatever makes the tests failed.
Maybe in the long run, we can have a reusable implementation. But that's for another day, I suppose :D
Docs look ok.
I think it would be nice if it could poll but set the default to not poll. Most of our pulling mechanisms support it so it would be odd if this one did not have that capability.
@mattdurham Can you help me understand setting the default to not poll for updates? Don't all of our components default to polling in the background?
My thought on the default to not polling is related to this comment Pricing per API calls for Secrets Manager is much expensive compared to AWS SSM Parameter Store and AWS S3 API. if it is expensive default to no but allow it to be poll. Also fine with the other direction, default to a low poll rate.
Comparing the pricing of AWS Secrets Manager to S3 does show that secrets manager is more expensive, but perhaps not significantly:
- AWS Secrets manager: $0.05 per 10,000 API calls
- S3: $0.004 per 10,000 GET calls
It will take 200,000 polls of secrets for it to cost $1 USD. If we set the polling interval to hourly, and you had 100 Alloy instances all running the same component, that would still only be 73,000 polls per month ($0.36).
It seems like the cost is reasonable enough if we used a 1h polling interval by default. If you're running Alloy in an environment where you need AWS secrets manager (likely Enterprise-like), running Alloy itself probably costs orders of magnitude more money than the secrets manager bill would be.
WDYT @mattdurham?