s3proxy Unable to create parent directories of …. when using local file system and deployment without docker

Hi all,

I was hoping this to be a basic error but I get this error message when using the application. Just for reference I'm using this as part of my test framework and I have a tool that connects to Kafka and then generates data into S3 but somehow the writing with this test framework is failing (I don't have the error writing to actual S3).

16:43:51.516 [S3Proxy-Jetty-29] DEBUG org.gaul.shaded.org.eclipse.jetty.server.HttpChannelState - thrownException s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0
java.lang.RuntimeException: java.io.IOException: Unable to create parent directories of build\s3proxy\buckets\testbucket\.mpus-b5e39a04-e2d0-4a4f-a106-8eee3021ea5e-avro\com.abc.bde.msg\schema_version=1.0.0\year=2022\month=07\day=23\com.abc.bde.msg+0+0000000000.avro-stub-d164dce5-ff4e-4ed9-b830-f270afea9590
	at com.google.common.base.Throwables.propagate(Throwables.java:241)
	at org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:806)
	at org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:533)
	at org.jclouds.blobstore.config.LocalBlobStore.initiateMultipartUpload(LocalBlobStore.java:820)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
	at com.sun.proxy.$Proxy50.initiateMultipartUpload(Unknown Source)
	at org.gaul.s3proxy.S3ProxyHandler.handleInitiateMultipartUpload(S3ProxyHandler.java:2190)
	at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:720)
	at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:77)
	at org.gaul.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.gaul.shaded.org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.gaul.shaded.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.gaul.shaded.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.gaul.shaded.org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Unable to create parent directories of build\s3proxy\buckets\testbucket\.mpus-b5e39a04-e2d0-4a4f-a106-8eee3021ea5e-avro\com.abc.bde.msg\schema_version=1.0.0\year=2022\month=07\day=23\com.abc.bde.msg+0+0000000000.avro-stub-d164dce5-ff4e-4ed9-b830-f270afea9590
	at com.google.common.io.Files.createParentDirs(Files.java:485)
	at org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:552)
	at org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:795)
	... 28 common frames omitted

Any idea what might be? I have the most basic configuration:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:9999
jclouds.provider=filesystem
jclouds.filesystem.basedir=build/s3proxy/buckets

My setup normally starts by having one bucket (which exists): build/s3proxy/buckets/testbucket

But everything else normally in S3 gets created automatically. I was looking maybe at some configuration that would handle the folder creation?

Jul 25 '22 07:07 cstmgl

I don't have a Windows instance so it is hard for me to debug this. But it appears that Java does not believe that:

build\s3proxy\buckets\testbucket\.mpus-b5e39a04-e2d0-4a4f-a106-8eee3021ea5e-avro\com.abc.bde.msg\schema_version=1.0.0\year=2022\month=07\day=23\com.abc.bde.msg+0+0000000000.avro-stub-d164dce5-ff4e-4ed9-b830-f270afea9590

exists despite calling File.mkdirs. Windows has some strange restrictions on file names so I might experiment with some simpler file name. Specifically it appears that Windows limits MAX_PATH to 260 characters so perhaps the full length is too long? Note that only some Windows APIs limit paths to this length; it is possible that S3Proxy should call java.nio.file.Files#createDirectories or some newer API.

Jul 25 '22 12:07 gaul

I'll give it a try today and see if I can make something out of it. If yes I'll update it back here.

Jul 25 '22 13:07 cstmgl

I'll give it a try today and see if I can make something out of it. If yes I'll update it back here.

I'm giving it a try but I was looking at the documentation and I think for what I want I did a massive miss understanding of the tool.

I think the S3ProxyRule gives me most of the things I need for my tests. https://github.com/gaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects#example

Jul 25 '22 21:07 cstmgl