AwsS3EnvironmentRepository does not comply with Git

[Husted-1]

Describe the bug Using version 3.1.9 due to restrictions, but looking in the code of AwsS3EnvironmentRepository I see the newest version has the same issue. Our configuration is sorted into folders for each application with the name of that given application. The implementation does not handle this like the Git version - tho I see that the newest can use folders, but with hardcoded "application" as the name of the file.

Sample Works in Git, but not in AwsS3EnvironmentRepository :

app1/app1-profile.yml app1/app1.yml

What looks to work in AwsS3EnvironmentRepository in 4.3.0-SNAPSHOT:

app1/application-profile.yml app1/application.yml

[ryanjbaxter]

Please provide a sample that reproduces the error with a supported release (4.1.x or greater).

[Husted-1]

This is from the newest AwsS3EnvironmentRepository where you generate the S3 key:

	private String buildObjectKeyPrefix(String application, String profile, String label) {
		StringBuilder objectKeyPrefix = new StringBuilder();
		if (!ObjectUtils.isEmpty(label)) {
			objectKeyPrefix.append(label).append(PATH_SEPARATOR);
		}
		objectKeyPrefix.append(application);
		if (this.useApplicationAsDirectory) {
			objectKeyPrefix.append(PATH_SEPARATOR).append("application");
		}
		if (!ObjectUtils.isEmpty(profile)) {
			objectKeyPrefix.append("-").append(profile);
		}
		return objectKeyPrefix.toString();
	}

	private void addPropertySource(Environment environment, String app, String profile, String label) {
		String objectKeyPrefix = buildObjectKeyPrefix(app, profile, label);
		S3ConfigFile s3ConfigFile = getS3ConfigFile(objectKeyPrefix);
		if (s3ConfigFile != null) {
			environment.setVersion(s3ConfigFile.getVersion());

			final Properties config = s3ConfigFile.read();
			config.putAll(serverProperties.getOverrides());
			PropertySource propertySource = new PropertySource("s3:" + objectKeyPrefix, config);
			if (LOG.isDebugEnabled()) {
				LOG.debug("Adding property source to environment " + propertySource);
			}
			environment.add(propertySource);
		}
	}

As one can see, it doesn't cover my provided example of what the git version can.

Code found here: https://github.com/spring-cloud/spring-cloud-config/blob/main/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepository.java

[ryanjbaxter]

A concrete sample that reproduces your use case would help here to make sure we are on the same page.

It sounds like what you are suggesting is that AWS S3 Environment Respository support the pattern matching functionality that we have in Git. https://docs.spring.io/spring-cloud-config/reference/server/environment-repository/git-backend.html#pattern-matching-and-multiple-repositories

[Husted-1]

Yes, it could be a good thing if one could move ones config to a new repository without needing to change it.

This is the setup from application.yml on the cloud config server:

spring:
  profiles:
    active: default
  cloud:
    config:
      server:
        defaultLabel: master
        git:
          uri: [email protected]:####/cloud-config.git
          searchPaths: '/*'
          cloneOnStart: false

File setup like this works in git, and native, but not in awsS3:

app1/app1-profile.yml
app1/app1.yml
application-profile.yml
application.yml

The code for awsS3 wants the files to be named "application" and will not accept "app1", as it is hardcoded, like I wrote.

[ryanjbaxter]

OK this is what I thought, this would be a worthwhile enhancement

[tomy8964]

Hi @ryanjbaxter! I'd like to take on this enhancement and contribute a PR for S3 pattern matching support, similar to the Git backend.
I'll start working on this and will share a draft or questions here if I run into issues. Can you assign this to me? Thanks!

[ryanjbaxter]

Great thanks!

[tomy8964]

Hi @ryanjbaxter,

I wanted to ask for your opinion regarding possible implementation strategies for improving S3 backend compatibility with the Git backend's searchPaths feature.

There are three approaches I'm considering:

1. Full Git-style searchPaths support, including wildcards (*, **)
This would require using S3's ListObjectsV2 API to match files for wildcards, which may cause increased API calls and cost, especially in buckets with many objects.

2. Only extend the useDirectoryLayout option
We could enhance useDirectoryLayout to also look for files like {application}/{application}-{profile}.yml, which would help many use cases but not all Git patterns.

3. Support Git-style searchPaths with placeholders only (no wildcards)
This approach would support placeholders like {application}, {profile}, {label} in the searchPaths, but not wildcards. We could then check for specific S3 keys using headObject, minimizing API calls.

Which approach do you think would be most appropriate for the project?
Option 3 seems like a good compromise between flexibility and S3 efficiency, but I'd love to hear your thoughts.

Thanks!

[ryanjbaxter]

I am inclined to say option 1 since this is an opt in feature. We would only incur the cost if they user uses such wildcards otherwise we continue to operate as normal, right?

[tomy8964]

Yes, that's correct! With this approach, the additional S3 ListObjectsV2 calls (and related costs) would only occur if the user actually configures searchPaths containing wildcards. Otherwise, for normal configurations without wildcards, the backend would continue to use efficient single-key lookups as before. I agree, option 1 provides maximum flexibility for those who need it, while still being cost-effective by default.

I'll proceed with implementing option 1, making sure it's opt-in and clearly documented so users understand the potential impact of wildcards on cost and performance. Thank you for the guidance!

[tomy8964]

Hi @ryanjbaxter, I’ve opened a PR #2958 that implements support for Git-style searchPaths in the S3 backend. Please have a look when you get a chance! 👍