bottlerocket icon indicating copy to clipboard operation
bottlerocket copied to clipboard
verified publisher icon bottlerocket-os

pubsys: update to AWS SDK Rust

Open jpculp opened this issue 3 years ago • 6 comments

Issue number:

#1968

Description of changes:

Replaces rusoto with aws-sdk-for-rust in pubsys.

Testing done:

  • [x] publish ami to us-west-2
  • [x] publish multiple amis with chained assumed roles (2 different accounts, 3 regions)
  • [x] grant ami access
  • [x] revoke ami access
  • [x] publish ssm parameters
  • [x] promote ssm parameters

TUF testing done: Thanks, @etungsten!

  • [x] use KMS key when creating TUF repo
  • [x] check TUF repo expirations
  • [x] validate TUF repo
  • [x] refresh TUF repo

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

jpculp avatar Sep 12 '22 22:09 jpculp

Fixed the result unwrapping issue while also improving how and where the profile gets loaded.

jpculp avatar Sep 14 '22 20:09 jpculp

  • Fixed missing region from chained configuration.
  • Removed the extraneous else's.

jpculp avatar Sep 16 '22 22:09 jpculp

  • Removed extraneous clones and references.
  • Set default log level for aws_config and aws_smithy to Warn.
  • Improved error handling and logging.

jpculp avatar Sep 19 '22 18:09 jpculp

Tested check-repo-expirations:

$ cargo make  -e BUILDSYS_VARIANT=aws-k8s-1.23 -e BUILDSYS_ARCH=x86_64 check-repo-expirations
[cargo-make] INFO - cargo make 0.35.15
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: check-repo-expirations
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: tuftool
[cargo-make] INFO - Running Task: publish-setup-tools
[cargo-make] INFO - Running Task: publish-setup
19:01:24 [INFO] Found infra config at path: /home/ec2-user/bottlerocket/Infra.toml
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: publish-tools
[cargo-make] INFO - Running Task: check-repo-expirations
[2022-09-19T19:01:26Z INFO  pubsys_config] Found infra config at path: /home/ec2-user/bottlerocket/Infra.toml
[2022-09-19T19:01:26Z INFO  pubsys::repo::check_expirations] Loaded TUF repo:   file:///home/ec2-user/bottlerocket/build/repos/default/bottlerocket-1.9.2-deea371c/aws-k8s-1.23/x86_64
[2022-09-19T19:01:26Z INFO  pubsys::repo::check_expirations] Root expiration:   2023-09-18 18:58:32 UTC
[2022-09-19T19:01:26Z INFO  pubsys::repo::check_expirations] Snapshot expiration:       2022-10-03 19:01:12.156262570 UTC
[2022-09-19T19:01:26Z INFO  pubsys::repo::check_expirations] Targets expiration:        2022-10-03 19:01:12.156262570 UTC
[2022-09-19T19:01:26Z INFO  pubsys::repo::check_expirations] Timestamp expiration:      2022-09-26 19:01:12.156262570 UTC
[2022-09-19T19:01:26Z INFO  pubsys::repo::check_expirations] Looking for metadata expirations happening from now to 2022-09-22 19:01:26.973914183 UTC
[cargo-make] INFO - Build Done in 4.72 seconds.

Tested validate-repo:

$ cargo make  -e BUILDSYS_VARIANT=aws-k8s-1.23 -e BUILDSYS_ARCH=x86_64 validate-repo
[cargo-make] INFO - cargo make 0.35.15
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: validate-repo
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: tuftool
[cargo-make] INFO - Running Task: publish-setup-tools
[cargo-make] INFO - Running Task: publish-setup
19:01:45 [INFO] Found infra config at path: /home/ec2-user/bottlerocket/Infra.toml
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: publish-tools
[cargo-make] INFO - Running Task: validate-repo
[2022-09-19T19:01:47Z INFO  pubsys_config] Found infra config at path: /home/ec2-user/bottlerocket/Infra.toml
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Loaded TUF repo: file:///home/ec2-user/bottlerocket/build/repos/default/bottlerocket-1.9.2-deea371c/aws-k8s-1.23/x86_64
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: migrate_v1.10.0_dns-settings.lz4
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: bottlerocket-aws-k8s-1.23-x86_64-v1.9.2.img.lz4
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: manifest.json
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: bottlerocket-aws-k8s-1.23-x86_64-1.9.2-deea371c-root.ext4.lz4
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: bottlerocket-aws-k8s-1.23-x86_64-1.9.2-deea371c-boot.ext4.lz4
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: bottlerocket-aws-k8s-1.23-x86_64-1.9.2-deea371c.img.lz4
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: aws-k8s-1.23-x86_64-kmod-kit-v1.9.2.tar.xz
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: bottlerocket-aws-k8s-1.23-x86_64-1.9.2-deea371c-data.img.lz4
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: bottlerocket-aws-k8s-1.23-x86_64-v1.9.2-data.img.lz4
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: bottlerocket-aws-k8s-1.23-x86_64-1.9.2-deea371c-root.verity.lz4
[2022-09-19T19:01:47Z INFO  pubsys::repo::validate_repo] Downloading target: migrate_v1.10.0_dns-settings-metadata.lz4
[cargo-make] INFO - Build Done in 4.25 seconds.

Tested refresh-repo with KMS key:

$ cargo make  -e BUILDSYS_VARIANT=aws-k8s-1.23 -e BUILDSYS_ARCH=x86_64 refresh-repo        
[cargo-make] INFO - cargo make 0.35.15
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: refresh-repo
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: tuftool
[cargo-make] INFO - Running Task: publish-setup-tools
[cargo-make] INFO - Running Task: publish-setup
19:05:28 [INFO] Found infra config at path: /home/ec2-user/bottlerocket/Infra.toml
[cargo-make] INFO - Running Task: publish-tools
[cargo-make] INFO - Running Task: refresh-repo
[2022-09-19T19:05:28Z INFO  pubsys_config] Found infra config at path: /home/ec2-user/bottlerocket/Infra.toml
[2022-09-19T19:05:28Z INFO  pubsys::repo::refresh_repo] Using repo expiration policy from path: /home/ec2-user/bottlerocket/tools/pubsys/policies/repo-expiration/2w-2w-1w.toml
[2022-09-19T19:05:28Z INFO  pubsys::repo::refresh_repo] Loaded TUF repo: file:///home/ec2-user/bottlerocket/build/repos/default/bottlerocket-1.9.2-deea371c-bak/aws-k8s-1.23/x86_64
[2022-09-19T19:05:28Z INFO  pubsys::repo] Setting non-root metadata expiration times:
        snapshot:  2022-10-03 19:05:28.522312124 UTC
        targets:   2022-10-03 19:05:28.522312124 UTC
        timestamp: 2022-09-26 19:05:28.522312124 UTC
[2022-09-19T19:05:28Z INFO  pubsys::repo::refresh_repo] Writing repo metadata to: /home/ec2-user/bottlerocket/build/repos/default/bottlerocket-1.9.2-deea371c/aws-k8s-1.23/x86_64
[cargo-make] INFO - Build Done in 2.40 seconds.

etungsten avatar Sep 19 '22 19:09 etungsten

  • Brought back stronger-typed regions and region_from_string().
  • Renamed aws: AwsConfig to pubsys_aws_config: PubsysAwsConfig to reduce confusion.
  • Kept arch as a stronger-type throughout.
  • Replaced ImageState ensure statement with match.
  • Replaced thread::spawn with tokio::runtime::Runtime::new().

jpculp avatar Sep 20 '22 22:09 jpculp

  • Changed credentials logic will run even if only a single role is specified.
  • Credential expirations can now be None.
  • Removed endpoints.

jpculp avatar Sep 21 '22 01:09 jpculp

  • Rewrote client.rs to return providers instead of credentials.
  • Removed returning unneccessary result-types (and their associated error contexts).

jpculp avatar Sep 22 '22 19:09 jpculp

Suppressed noisy [2022-09-22T20:01:26Z INFO tracing::span] assume_role; log.

jpculp avatar Sep 22 '22 20:09 jpculp