tycho ProvisionException already exists in repository

In a parallelized build I see the following exception

Caused by: org.eclipse.equinox.p2.core.ProvisionException: Artifact canonical: osgi.bundle,org.eclipse.core.databinding.property,1.9.0.v20210619-1129 already exists in repository file:<path>/.mvn/
    at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl$AddingArtifactSink.<init> (ArtifactRepositoryBaseImpl.java:402)
    at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl.internalNewAddingArtifactSink (ArtifactRepositoryBaseImpl.java:382)
    at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl.newAddingArtifactSink (ArtifactRepositoryBaseImpl.java:376)
    at org.eclipse.tycho.repository.local.MirroringArtifactProvider.downloadCanonicalArtifact (MirroringArtifactProvider.java:325)
    at org.eclipse.tycho.repository.local.MirroringArtifactProvider.ensureArtifactIsPresentInCanonicalFormat (MirroringArtifactProvider.java:340)
    at org.eclipse.tycho.repository.local.MirroringArtifactProvider.makeLocallyAvailable (MirroringArtifactProvider.java:202)
    at org.eclipse.tycho.repository.local.MirroringArtifactProvider.getArtifactFile (MirroringArtifactProvider.java:132)
    at org.eclipse.tycho.repository.p2base.artifact.provider.CompositeArtifactProvider.getArtifactFile (CompositeArtifactProvider.java:87)
    at org.eclipse.tycho.p2.target.TargetPlatformBaseImpl.getLocalArtifactFile (TargetPlatformBaseImpl.java:154)
    at org.eclipse.tycho.p2.resolver.DefaultP2ResolutionResult.lambda$2 (DefaultP2ResolutionResult.java:89)
    at org.eclipse.tycho.p2.resolver.DefaultP2ResolutionResultEntry.getLocation (DefaultP2ResolutionResultEntry.java:73)
    at org.eclipse.tycho.p2.resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$2 (P2DependencyResolver.java:479)
    at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2 (ArtifactCollection.java:66)
    at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:62)
    at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:121)
    at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:62)
    at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:121)
    at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:62)
    at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies (MavenDependencyInjector.java:142)
    at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency (MavenDependencyInjector.java:134)
    at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies (MavenDependencyInjector.java:66)
    at org.eclipse.tycho.p2.resolver.P2DependencyResolver.injectDependenciesIntoMavenModel (P2DependencyResolver.java:499)
    at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:178)
    at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.lambda$resolveProjects$0 (TychoMavenLifecycleParticipant.java:157)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept (ForEachOps.java:183)
    at java.util.stream.SpinedBuffer$1Splitr.forEachRemaining (SpinedBuffer.java:364)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:484)
    at java.util.stream.ForEachOps$ForEachTask.compute (ForEachOps.java:290)
    at java.util.concurrent.CountedCompleter.exec (CountedCompleter.java:746)
    at java.util.concurrent.ForkJoinTask.doExec (ForkJoinTask.java:290)
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec (ForkJoinPool.java:1020)
    at java.util.concurrent.ForkJoinPool.scan (ForkJoinPool.java:1656)
    at java.util.concurrent.ForkJoinPool.runWorker (ForkJoinPool.java:1594)
    at java.util.concurrent.ForkJoinWorkerThread.run (ForkJoinWorkerThread.java:177)

FYI @merks seems similar to what I recently encountered with Oomph

May 29 '22 07:05 laeubi

Sounds a bit like #663, but you probably know better if there is any relation between them.

May 29 '22 12:05 Bananeweizen

No this is sadly something different, this error is emitted by P2, while the other is emitted by the OS.

May 30 '22 05:05 laeubi

Yes, I very occasionally see this problem in Oomph as well. But it's never possible to reproduce because if you try again, it succeeds. I suppose one could manually reproduce it by placing the artifact that will be download at the location where it will end up. Instead of failing though, p2 might handle it more gracefully and produce only a warning/log rather than fail the overall process; but should p2 overwrite the artifact or leave it as is?

May 30 '22 06:05 merks

p2 might handle it more gracefully and produce only a warning/log rather than fail the overall process; but should p2 overwrite the artifact or leave it as is?

That's the question, I wonder what one would gain here to throw errors, at least at Tycho there is even code to first remove the descriptor and add it afterwards if it already exits, but still there is of course a window where a race could happen. I think the initial idea was that there should be no intermediate state (e.g. A descriptor is visible to other threads but its contents is overwritten then), tycho uses a "downloadlock" for this but it seems this is not enough here.

May 30 '22 06:05 laeubi

should p2 overwrite the artifact or leave it as is?

I believe the fact that there is no clear answer what's best is the reason why an exception was the chosen behavior: at least one can't blame p2 for taking the wrong decision (ie installing an unexpected artifact, or corrupting an existing installation).

Isn't the issue that multiple thread try to download the exact same file at the same time? If so, I imagine we can protect the code against that by putting some MirrorArtifact on hold if they happen to target the same destination as one currently running. Then upon completion of first mirror action, the next MirrorArtifact for this destination would check the target file signature/size and compare it to the ones expexted by p2 and continue if they are a match (which I think is most often the case).

May 30 '22 07:05 mickaelistria

No, I don't think multiple threads are trying to download the same artifact. In general, the set of artifact requests is collected and requests are distributed across threads, but that's a comment about how p2 (and Oomph) use a transaction to update a profile. I don't know what Tycho is doing in this regard...

I like the idea of comparing the artifacts. Obviously if they are equal such an issue could be quietly ignored...

May 30 '22 07:05 merks

The problem is, to compare them one needs to download them, but as the download is rejected because the artifact already exits ... Anyways I already added support to Tycho to use the sha1 + md5 to decide if an artifact needs to be downloaded again if the properties change.

Also, Tycho here only emulates P2 (the error is from pure Tycho code!) and already has a "download lock" so the problem is probably more that the artifactrepositorymanager making the same thing available to different parts of the code and then it could be happen that things are not in sync anymore.

May 30 '22 07:05 laeubi

We're experiencing the same on tycho 3.0.4. Sometimes the build is failing due to the below exception. Running it multiple times again later, works just fine.

Kind regards,, Richard

Caused by: org.eclipse.equinox.p2.core.ProvisionException: Artifact canonical: osgi.bundle,jcl.over.slf4j,1.7.32 already exists in repository file:/data/buildserver/jenkins/maven/../../maven/product.feature/myfeature/ at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl$AddingArtifactSink.<init> (ArtifactRepositoryBaseImpl.java:402) at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl.internalNewAddingArtifactSink (ArtifactRepositoryBaseImpl.java:382) at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl.newAddingArtifactSink (ArtifactRepositoryBaseImpl.java:376) at org.eclipse.tycho.repository.local.MirroringArtifactProvider.downloadCanonicalArtifact (MirroringArtifactProvider.java:313) at org.eclipse.tycho.repository.local.MirroringArtifactProvider.ensureArtifactIsPresentInCanonicalFormat (MirroringArtifactProvider.java:333)

Jul 17 '23 07:07 richardgroote

I encounter this issue as well with Tycho 4.0.2 frequently and first assumed it is a regression in the new version (but this issue proofs that wrong). We recently also changed to parallel builds so maybe the parallel resolution of dependencies is causing this problem? The artifact is actually not new and we use a shared .m2 cache (but since that is an artifact from a p2-repo it is probably not cached forever) and AFAICT there was no other job running at the time. So my initial guess is that the parallel resolution is interfering with itself. Wouldn't it be possible to synchronize the mirror of the same artifact? From the metadata it should be possible to determine if it is the same or not.

 [ERROR] foo.bar: Error while mirroring artifact osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 to the local Maven repositoryArtifact canonical: osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 already exists in repository file:/home/build/.m2/repository/
 java.lang.RuntimeException: foo.bar: Error while mirroring artifact osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 to the local Maven repositoryArtifact canonical: osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 already exists in repository file:/home/build/.m2/repository/
     at io.takari.maven.builder.smart.SmartBuilderImpl.buildProject (SmartBuilderImpl.java:214)
     at io.takari.maven.builder.smart.SmartBuilderImpl$ProjectBuildTask.run (SmartBuilderImpl.java:81)
     at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
     at java.util.concurrent.FutureTask.run (FutureTask.java:264)
     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
     at java.lang.Thread.run (Thread.java:833)
 Caused by: org.eclipse.tycho.p2.repository.MirroringArtifactProvider$MirroringFailedException: Error while mirroring artifact osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 to the local Maven repositoryArtifact canonical: osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 already exists in repository file:/home/build/.m2/repository/
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeLocallyAvailable (MirroringArtifactProvider.java:221)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.getArtifactFile (MirroringArtifactProvider.java:137)
     at org.eclipse.tycho.p2.repository.CompositeArtifactProvider.getArtifactFile (CompositeArtifactProvider.java:89)
     at org.eclipse.tycho.p2resolver.TargetPlatformBaseImpl.getLocalArtifactFile (TargetPlatformBaseImpl.java:169)
     at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult.lambda$addArtifact$2 (DefaultP2ResolutionResult.java:90)
     at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResultEntry.getLocation (DefaultP2ResolutionResultEntry.java:73)
     at org.eclipse.tycho.p2resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$6 (P2DependencyResolver.java:383)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2 (ArtifactCollection.java:69)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies (MavenDependencyInjector.java:216)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency (MavenDependencyInjector.java:180)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies (MavenDependencyInjector.java:82)
     at org.eclipse.tycho.p2resolver.P2DependencyResolver.injectDependenciesIntoMavenModel (P2DependencyResolver.java:409)
     at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:155)
     at org.eclipse.tycho.core.maven.TychoProjectExecutionListener.beforeProjectLifecycleExecution (TychoProjectExecutionListener.java:108)
     at org.apache.maven.lifecycle.internal.CompoundProjectExecutionListener.beforeProjectLifecycleExecution (CompoundProjectExecutionListener.java:42)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:103)
     at io.takari.maven.builder.smart.SmartBuilderImpl.buildProject (SmartBuilderImpl.java:209)
     at io.takari.maven.builder.smart.SmartBuilderImpl$ProjectBuildTask.run (SmartBuilderImpl.java:81)
     at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
     at java.util.concurrent.FutureTask.run (FutureTask.java:264)
     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
     at java.lang.Thread.run (Thread.java:833)
 Caused by: org.eclipse.equinox.p2.core.ProvisionException: Artifact canonical: osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 already exists in repository file:/home/build/.m2/repository/
     at org.eclipse.tycho.p2.repository.ArtifactRepositoryBaseImpl$AddingArtifactSink.<init> (ArtifactRepositoryBaseImpl.java:412)
     at org.eclipse.tycho.p2.repository.ArtifactRepositoryBaseImpl.internalNewAddingArtifactSink (ArtifactRepositoryBaseImpl.java:392)
     at org.eclipse.tycho.p2.repository.ArtifactRepositoryBaseImpl.newAddingArtifactSink (ArtifactRepositoryBaseImpl.java:386)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.downloadCanonicalArtifact (MirroringArtifactProvider.java:338)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.ensureArtifactIsPresentInCanonicalFormat (MirroringArtifactProvider.java:353)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeLocallyAvailable (MirroringArtifactProvider.java:215)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.getArtifactFile (MirroringArtifactProvider.java:137)
     at org.eclipse.tycho.p2.repository.CompositeArtifactProvider.getArtifactFile (CompositeArtifactProvider.java:89)
     at org.eclipse.tycho.p2resolver.TargetPlatformBaseImpl.getLocalArtifactFile (TargetPlatformBaseImpl.java:169)
     at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult.lambda$addArtifact$2 (DefaultP2ResolutionResult.java:90)
     at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResultEntry.getLocation (DefaultP2ResolutionResultEntry.java:73)
     at org.eclipse.tycho.p2resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$6 (P2DependencyResolver.java:383)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2 (ArtifactCollection.java:69)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies (MavenDependencyInjector.java:216)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency (MavenDependencyInjector.java:180)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies (MavenDependencyInjector.java:82)
     at org.eclipse.tycho.p2resolver.P2DependencyResolver.injectDependenciesIntoMavenModel (P2DependencyResolver.java:409)
     at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:155)
     at org.eclipse.tycho.core.maven.TychoProjectExecutionListener.beforeProjectLifecycleExecution (TychoProjectExecutionListener.java:108)
     at org.apache.maven.lifecycle.internal.CompoundProjectExecutionListener.beforeProjectLifecycleExecution (CompoundProjectExecutionListener.java:42)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:103)
     at io.takari.maven.builder.smart.SmartBuilderImpl.buildProject (SmartBuilderImpl.java:209)
     at io.takari.maven.builder.smart.SmartBuilderImpl$ProjectBuildTask.run (SmartBuilderImpl.java:81)
     at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
     at java.util.concurrent.FutureTask.run (FutureTask.java:264)
     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
     at java.lang.Thread.run (Thread.java:833)

Sep 11 '23 09:09 HannesWell