AWS multi region infrastructure improvements

[amirbenun]

Background Today we use a multi-region wrapper to access AWS multi-regions with simple go functions, this implementation lies on the infrastructure and is being used by different fetchers and different use cases (vuln-mgm / cis_aws).

Motivation We can improve the AWS multi-region infrastructure in a way that will let us consume it easily with more use cases. One major issue is that by default we always initialize the multi-region map with all the regions while in vuln-mgmt we can only use the default region. API example that will support the suggestions:

type CrossRegionFactory[T any] interface {
    NewMultiRegionClients(cfg awssdk.Config, factory func(cfg awssdk.Config) T, log *logp.Logger) CrossRegionFetcher[T]
    WithDefaultRegion()
    WithAllRegions(client DescribeCloudRegions)
}

Another suggestion I had, today, every time we use the multi-region provider we need to access its actual clients where instead I envision an implementation where the clients are private objects that should not be accessed directly. API example that is taken from https://github.com/elastic/cloudbeat/pull/654/files#diff-286aa8e2b44cb12d1ca095e4ee878de54159f0973fe827bac1fa075ae5cc4179:

type crossRegionProvider struct {
	CrossRegionFetcher awslib.CrossRegionFetcher[ElasticCompute]
}

func (c *crossRegionProvider) DescribeNetworkAcl(ctx context.Context) ([]awslib.AwsResource, error) {
	return c.CrossRegionFetcher.Fetch(func(ec ElasticCompute) ([]awslib.AwsResource, error) {
		return ec.DescribeNetworkAcl(ctx)
	})
}

Would like to hear your thoughts @uri-weisman @olegsu @jeniawhite

[olegsu]

Thank you, @amirbenun

One major issue is that by default we always initialize the multi-region map with all the regions while in vuln-mgmt we can only use the default region.

Does this seem like something that does not match for multi-region, am I wrong? I understand the desire to have the region be agnostic, and yet sometimes we need to know what is the region.

[amirbenun]

Maybe the name multi-region is confusing here. The idea suggests that we could have a single interface that supports all sorts of operations regardless of the region we want to use. Using the factory, you will have to declare what regions are relevant for your flow (all, single, default). Next, when instantiating the provider, the fetch operations on the providers will look exactly the same. All the code that iterates through different regions will have to be written once. So it's not necessarily multi-region but more like any-region.

[uri-weisman]

Thanks for raising this up @amirbenun.

IMO, using this layer for setting a single client might be overcomplicating things. I do agree with you that the abstraction layer adds elegancy to our code, however, coupling the providers to this interface adds complexity as in some providers there is a mix of regional/global api calls (e.g s3).