eclipse-tycho
SharedHttpCacheStorage failes to create directory with IOException for very long named cache entry
I have a case where the tycho transport fails with an IOException because it cannot create the directory to store the cached file.
The fetch url used to construct the cache path is rather long (over 2000 chars total, with the largest single segment being 1342 chars long) so it may be that the filename is too long for the filesystem.
This error is reproduced with 5.0.0-SNAPSHOT and the stacktrace is listed bellow.
Caused by: java.io.IOException: Cannot create directory '<REDACTED>'. at org.apache.commons.io.FileUtils.mkdirs (FileUtils.java:2289) at org.apache.commons.io.FileUtils.forceMkdir (FileUtils.java:1376) at org.eclipse.tycho.p2maven.transport.SharedHttpCacheStorage$CacheLine.updateHeader (SharedHttpCacheStorage.java:409) at org.eclipse.tycho.p2maven.transport.SharedHttpCacheStorage$CacheLine.fetchFile (SharedHttpCacheStorage.java:263) at org.eclipse.tycho.p2maven.transport.SharedHttpCacheStorage$2.getCacheFile (SharedHttpCacheStorage.java:133) at org.eclipse.tycho.p2maven.transport.SharedHttpCacheStorage$CacheLine.getFile (SharedHttpCacheStorage.java:311) at org.eclipse.tycho.p2maven.transport.SharedHttpCacheStorage$2.getCacheFile (SharedHttpCacheStorage.java:141) at org.eclipse.tycho.p2maven.transport.HttpTransportProtocolHandler.getFile (HttpTransportProtocolHandler.java:50) at org.eclipse.tycho.p2maven.transport.TychoRepositoryTransport.stream (TychoRepositoryTransport.java:141) at org.eclipse.tycho.p2maven.transport.TychoRepositoryTransport.download (TychoRepositoryTransport.java:104) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.downloadArtifact (SimpleArtifactRepository.java:737) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.downloadArtifact (SimpleArtifactRepository.java:659) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.getArtifact (SimpleArtifactRepository.java:796) at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromOneMirror (RepositoryArtifactProvider.java:243) at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromAnyMirror (RepositoryArtifactProvider.java:223) at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider$1.perform (RepositoryArtifactProvider.java:198) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.getArtifact (SimpleArtifactRepository.java:778) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.getArtifacts (SimpleArtifactRepository.java:858) at org.eclipse.tycho.p2.repository.LazyArtifactRepository.getArtifacts (LazyArtifactRepository.java:102) at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromAnyChildRepository (RepositoryArtifactProvider.java:210) at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromAnyFormatAvailableInRepository (RepositoryArtifactProvider.java:177) at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromAnySource (RepositoryArtifactProvider.java:163) at org.eclipse.tycho.p2.repository.CompositeArtifactProviderBaseImpl.getArtifact (CompositeArtifactProviderBaseImpl.java:55) at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.downloadCanonicalArtifact (MirroringArtifactProvider.java:339) at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.downloadMostSpecificNeededFormatOfArtifact (MirroringArtifactProvider.java:332) at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.downloadArtifact (MirroringArtifactProvider.java:317) at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeOneFormatLocallyAvailable (MirroringArtifactProvider.java:256) at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeLocallyAvailable (MirroringArtifactProvider.java:207) at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.getArtifactFile (MirroringArtifactProvider.java:138) at org.eclipse.tycho.p2.repository.CompositeArtifactProvider.getArtifactFile (CompositeArtifactProvider.java:89) at org.eclipse.tycho.p2resolver.TargetPlatformBaseImpl.getLocalArtifactFile (TargetPlatformBaseImpl.java:169) at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult.lambda$addArtifact$2 (DefaultP2ResolutionResult.java:90) at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResultEntry.getLocation (DefaultP2ResolutionResultEntry.java:73) at org.eclipse.tycho.p2resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$6 (P2DependencyResolver.java:403) at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2 (ArtifactCollection.java:69) at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72) at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148) at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72) at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148) at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72) at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies (MavenDependencyInjector.java:255) at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency (MavenDependencyInjector.java:219) at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies (MavenDependencyInjector.java:120) at org.eclipse.tycho.p2resolver.P2DependencyResolver.injectDependenciesIntoMavenModel (P2DependencyResolver.java:429) at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:100) at org.eclipse.tycho.core.maven.TychoProjectExecutionListener.beforeProjectLifecycleExecution (TychoProjectExecutionListener.java:108) at org.apache.maven.lifecycle.internal.CompoundProjectExecutionListener.beforeProjectLifecycleExecution (CompoundProjectExecutionListener.java:42) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:103) at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:193) at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:180) at java.util.concurrent.FutureTask.run (FutureTask.java:317) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:572) at java.util.concurrent.FutureTask.run (FutureTask.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642) at java.lang.Thread.run (Thread.java:1583)
So digging a little deeper. The offending url segment is in the form &X-Amz-Security-Token=<some-token-data> for the S3 bucket that backs our internal repo's storage.
Modifying Tycho to split the cleanPath in getCacheLine such that no path segment is longer than 128 chars fixes the issue.
@bengaineyarm thanks for the analysis, passing a Security-Token as part of the URL looks like a security risk to me (as you see it will be recorded in the cache and probably also in logfiles as well as in your SCM). Is there an alternative way to pass such sensitive information?
Modifying Tycho to split the cleanPath in getCacheLine such that no path segment is longer than 128 chars fixes the issue.
Do you like to propose a PR?
@laeubi I believe the token is transient (something like Artifactory forwarding to the underlying S3 bucket rather than serving the file its self), but i'll raise it with our infrastructure people.
Do you like to propose a PR?
Let me see if we (arm) have contributor agreement accepted first...