Publishing order to Maven Central seems wrong

[jonatan-ivanov]

trafficstars

Describe the bug

Every time there is an SDK release to Maven Central, people who depend on latest version will have a broken build since during the release the order of releasing the artifacts is wrong.

If there are artifacts A and B and A depends on B then B should be published first then A. This is not the case with the AWS SDK. Here's a real life example with Micrometer's build which depends on software.amazon.awssdk:cloudwatch which depends on software.amazon.awssdk:sdk-core (latest). Since cloudwatch:2.20.34 was published first, the build system is looking for its dependency sdk-core:2.20.34 which is not in Maven Central yet:

* What went wrong:
Execution failed for task ':micrometer-registry-cloudwatch2:compileJava'.
> Could not resolve all files for configuration ':micrometer-registry-cloudwatch2:compileClasspath'.
   > Could not find sdk-core-2.20.34.jar (software.amazon.awssdk:sdk-core:2.20.34).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/software/amazon/awssdk/sdk-core/2.20.34/sdk-core-2.20.34.jar

Expected Behavior

Not having a state in Maven Central that is incomplete/broken in terms of dependencies. Take dependency ordering into consideration when publishing to Maven Central so that people's builds will not be broken (or stage everything first and release one).

Current Behavior

Right now during releases, Maven Central is incomplete/broken in terms of dependencies. Dependency ordering is not taken into consideration when publishing to Maven Central and people's builds are broken.

Reproduction Steps

This is tricky since this only happens during releases (though they are quite frequent in case of the AWS SDK) but during a release, try to build a project that has a dependency from the AWS SDK that has another dependency which is published later than the one you depend on. Example, try to build a project with this (repo is mavenCentral()):

implementation 'software.amazon.awssdk:cloudwatch:latest.release'

This will fail between software.amazon.awssdk:cloudwatch is already published but software.amazon.awssdk:sdk-core hasn't been. This can take quite a long time actually.

Possible Solution

Take dependency ordering into consideration when publishing to Maven Central so that people's builds will not be broken (or stage everything first and release one).

Additional Information/Context

Example CircleCI workflow and Build Scan result.

AWS Java SDK version used

irrelevant (release.latest) but this is happening right now with 2.20.34

JDK version used

irrelevant (openjdk 20 2023-03-21)

Operating System and version

irrelevant (MacOS 13.2.1)

[debora-ito]

I'm sorry but we don't have control over the order of the modules when they get published to Maven Central, nor the time it takes for all the modules to be available.

The recommendation is to not use latest but pin a specific version of the SDK dependency.

[jonatan-ivanov]

@debora-ito I'm not sure I understand this answer, you don't have ownership over the build of the AWS SDK? We have a good reason for depending on the latest version, also, if I would pin to a certain version which would be the latest, I would have the same issue, depending on latest is not the issue but the publishing mechanism.

[debora-ito]

We don't have control over the order in which the modules are published. I can't tell Maven Central to publish aws-core first, then sdk-core, then apache-client, then cloudwatch etc.

[jonatan-ivanov]

Are you staging and closing the artifacts one-by-one or do you upload everything to the staging repo and close it in one step?

[hironytic]

I'm not sure the publishing order is related or not, sdk-core 2.20.34 seems to be something wrong anyway. sdk-core-2.20.34.jar is listed in https://repo.maven.apache.org/maven2/software/amazon/awssdk/sdk-core/2.20.34/, but it links to "404 Not Found"

Though this maybe a problem of Maven Central side, all of your sdk 2.20.34 which eventually depends on sdk-core 2.20.34, seems to trigger a build failure. Such as:

• cloudWatch 2.20.34, depends on aws-query-protocol 2.20.34 and it depends on sdk-core 2.20.34.
• s3 2.20.34, depends on aws-xml-protocol 2.20.34 and it depends on sdk-core 2.20.34.

[a3linux]

Hit the same errors here.

[ersione]

I'm not sure the publishing order is related or not, sdk-core 2.20.34 seems to be something wrong anyway. sdk-core-2.20.34.jar is listed in https://repo.maven.apache.org/maven2/software/amazon/awssdk/sdk-core/2.20.34/, but it links to "404 Not Found"

Though this maybe a problem of Maven Central side, all of your sdk 2.20.34 which eventually depends on sdk-core 2.20.34, seems to trigger a build failure. Such as:

• cloudWatch 2.20.34, depends on aws-query-protocol 2.20.34 and it depends on sdk-core 2.20.34.
• s3 2.20.34, depends on aws-xml-protocol 2.20.34 and it depends on sdk-core 2.20.34.

Same issue with me.

[scarytom]

I think 2.20.34 was only partially released, which is a different issue to this one. I raised https://github.com/aws/aws-sdk-java-v2/issues/3866

[debora-ito]

Now, 2.20.34 being partially released seems a problem.

As of now, https://repo.maven.apache.org/maven2/software/amazon/awssdk/sdk-core/2.20.34/ works fine for me, but 2.20.34 is not listed here - https://mvnrepository.com/artifact/software.amazon.awssdk/sdk-core

Since this is in reference to a Java 2.x version, let's keep track of this issue in one place: https://github.com/aws/aws-sdk-java-v2/issues/3866. I'll add any updates over there.

[jonatan-ivanov]

Now, 2.20.34 being partially released seems a problem.

@debora-ito The two issues are different. This one happens quite frequently (every release I guess), this is definitely not the same issue of 2.20.34 being partially released (but I guess this time it is hiding behind it). Could you please reopen?

(Btw mvnrepository is not Maven Central, it was created by @frodriguez and it is not the official repo of Maven, so I think it is not as big of a concern that artifacts are not there.)

[github-actions[bot]]

COMMENT VISIBILITY WARNING

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[jonatan-ivanov]

Thank you, I think it is totally possible that this is caused by Maven Central every time and not the build setup of the AWS SDK (I bump into frequently) but investigating this would be great since this is an issue regardless of the root cause.

I have a few ideas/question to try to narrow this down:

Are you staging and closing the artifacts one-by-one or do you upload everything to the staging repo and close it in one step or you do something else?

Which host are you using for publishing? Are you suing the new one (s01.oss.sonatype.org) or the legacy one (oss.sonatype.org)? See OSSRH docs.

[zacthompson]

One idea would be to release the bom separately, after all the rest of the artifacts have been published in central.

[debora-ito]

Moving this to the Java SDK 2.x repo.

[debora-ito]

Discussed this with the team.

All artifacts are uploaded to Sonatype in one atomic operation. After they are released, they are under the Sonatype realm. We don't have plans to break the upload in two or more parts, but even if we did it's unclear if it would have the desired effect, since there's no guarantee the first batch will become available before the second.

[mmarston]

I've reported this as an issue to Sonatype too: https://issues.sonatype.org/browse/MVNCENTRAL-7999

[jonatan-ivanov]

@debora-ito Thank you very much looking into this. Could you please answer one last question? Which host are you using for publishing? Are you suing the new one (s01.oss.sonatype.org) or the legacy one (oss.sonatype.org)? See OSSRH docs.

[debora-ito]

@jonatan-ivanov neither.

https://github.com/aws/aws-sdk-java-v2/blob/299cbb075f7b2134d862d0f5944903ef42c8a0dd/pom.xml#L721

[jonatan-ivanov]

@debora-ito This is great, having a dedicated endpoint should make this easier to investigate for Sonatype and should be more performant.

[millems]

Sonatype made some changes related to cache purging around the time that we started seeing this issue. They have reverted those changes. We'll be working to verify on our end that this has resolved this issue, but can you report back if you see this issue again?

[jonatan-ivanov]

@millems I'm not sure when you started seeing this issue but to me it is not new. I have records that this happened quite a lot throughout 2022, my earliest record is 2022-02-22 but I think this happened in 2021 too I guess at every release.

[paulbakker]

+1 on the issue having been there for quite some time, this is not just recent.

[millems]

Oh, odd. We have gotten 3 reports of this issue in the last 2 week, and had never seen this report before then. We assumed this recently started happening or just got considerably worse, but maybe it's just a coincidence that we got so many reports at once.

Do you have records to see if it has gotten worse lately, or has it always been as bad as it has been recently?

[paulbakker]

I don't have records to back up my claims; it's anecdotal, but we've seen the same issue for at least a year. Previously I shrugged it off as being an isolated incident, but it seems to be a pattern.

[jonatan-ivanov]

Do you have records to see if it has gotten worse lately, or has it always been as bad as it has been recently?

Not really, the records I have is basically my notes or other people reporting it. I don't have proper metrics about these so I can't tell for sure but I can tell some biased feelings/opinion:

I think in the past (and still) I bumped into this less frequently than monthly. Also, I feel it is not happening more frequently but when it happens, it takes more time than it used to that might explain getting more complaints. I remember that when I bumped into this in the past, the issue went away in a few hours (less than 3 I guess). When I opened this issue (which might be two issues together), it took more than 24 hours. (Also, I'm biased because I've been building less in the past 1-2 month so maybe even if this happens more frequently I would not notice).

[dagnir]

@paulbakker @jonatan-ivanov We've been discussing this issue with Sonatype here: https://issues.sonatype.org/browse/MVNCENTRAL-7999 and they have made changes to their CDN. Have you seen this issue recently?

[jonatan-ivanov]

No, I haven't seen this issue since I opened this one.

[paulbakker]

We ran into it again today :(

[jonatan-ivanov]

Fyi, it seems it is happening right now (there is a release ongoing):

> Could not resolve software.amazon.awssdk:cloudwatch:2.20.148.
   > Could not parse POM https://repo.maven.apache.org/maven2/software/amazon/awssdk/cloudwatch/2.20.148/cloudwatch-2.20.148.pom
      > Could not find software.amazon.awssdk:services:2.20.148.