feat(aws_s3 source): separate s3 and sqs auth

[sonnens]

Summary

add an s3_auth config option, using different credentials for polling SQS and fetching files from S3

Change Type

• [ ] Bug fix
• [x] New feature
• [ ] Non-functional (chore, refactoring, docs)
• [ ] Performance

Is this a breaking change?

• [ ] Yes
• [X] No

How did you test this PR?

a different role authorized to poll SQS and S3

Does this PR include user facing changes?

• [X] Yes. Please add a changelog fragment based on our guidelines.
• [ ] No. A maintainer will apply the "no-changelog" label to this PR.

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• The CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
    • ./scripts/check_changelog_fragments.sh
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

References

• Closes: https://github.com/vectordotdev/vector/issues/23078

[milas]

Had this same use case come up and found this PR, so cherry-picked it - thanks 👍🏻

fwiw I do think that having auth remain as the S3 auth and making sqs.auth the "override" would be more consistent, as the other SQS config is already in under the sqs key: https://vector.dev/docs/reference/configuration/sources/aws_s3/#sqs

[sonnens]

Had this same use case come up and found this PR, so cherry-picked it - thanks 👍🏻

fwiw I do think that having auth remain as the S3 auth and making sqs.auth the "override" would be more consistent, as the other SQS config is already in under the sqs key: https://vector.dev/docs/reference/configuration/sources/aws_s3/#sqs

on precedence:

sqs: auth: ... and no auth clause on the outer config I think is pretty intuitive, "assume this role for the SQS step, use default instance / env creds for the S3 step"

The opposite of that, "assume a role for the S3 step, not the SQS step" would, I guess need an auth: default option and, if not present, use the outer auth config?

more concretely,

Both the SQS & S3 calls assume the role

aws_s3:
  type: aws_s3
  auth:
    assume_role: <whatever>
  sqs:
   queue_url: ...

Only the SQS call assumes the role , the S3 call uses the default system credentials:

aws_s3:
  type: aws_s3
  sqs:
   queue_url: ...
   auth:
    assume_role: <whatever>
  ...

Only the S3 call assumes the role , the SQS call uses the default system credentials:

aws_s3:
  type: aws_s3
  auth:
    assume_role: <whatever>
  sqs:
   queue_url: ...
   auth: default
  ...

there's one use case that I can't think of a sensible way to articulate and that's AssumeRole chaining , "first assume role X, then as role X, assume role Y to do the other step" so probably just ... don't do that ? or launch vector as the role you want it to be in the first place

[sonnens]

@thomasqueirozb & @pront

took the suggestions, now SQS's auth section set to a string literal falls back to the system / AWS SDK strategy and there's no s3_auth top level section. Probably makes more sense

[sonnens]

@pront anything I need to do for this?

[pront]

@pront anything I need to do for this?

No, thanks @sonnens. We will have to do another review soon.

[marnao]

just want to say thanks for working on this... we have the same requirement. looking forward to this feature.