S3Mock
S3Mock copied to clipboard
adobe
S3Mock does not start if data is in rootfolder from a previous run
Split from #786 Requested by @masterkain
Until 2.5.0, it was possible to run S3Mock with retainFilesOnExit=true, stop S3Mock and then start it again to serve those objects (as long as it was not a MultipartUpload, which were always mapped in memory, with only the binaries stored in memory).
With 2.5.0, all objects are stored in a directory that does not reflect their key, since that has always been problematic (encoding and decoding the key many times over depending on the use-case is error-prone and entirely unnecessary if the key is used internally only to find the correct object UUID, and with that, the folder all data is persisted in) While this mapping is persisted to disk, we also use the UUID to lock file system access internally, and this mapping is entirely in-memory.
While retainFilesOnExit=true is mainly meant to help with debugging to see if all data was persisted correctly, it could be possible to refactor S3Mock yet again to read all mappings on startup and serve all object requests as if it had been running all the time.
When doing this, the Multipart mapping should be persisted to disk as well to support the on/off use-case for all APIs.
from #786
I have some use cases for s3mock, I use it in specs (no volume), as a local s3 compatible container for storing development uploads (no volume) and on kubernetes as an s3 layer to store cached data between builds (volume).
On kubernetes I have it on auto image update and retainFilesOnExit, the problem did start with the 2.5.x series I believe. So it's not data that I should persist long term but I kinda expect to have retainFilesOnExit at least not to crash on startup. I might even be fine with filesystem getting wiped on every version update, but at least not crash this way especially given I had no troubles in the past (as long as at least with the same version something is persisted between restarts).
Even if it is not possible to start up the application with local files in place (for whatever reason, for example because files are unreadable or not found, or maybe because this issue can't be implemented), S3Mock should fail with a human readable error instead of throwing a cryptic error at runtime like it does right now:
│ 2022-09-09 18:31:00.476 INFO 1 --- [ main] c.a.testing.s3mock.S3MockApplication : Starting S3MockApplication using Java 17.0.4.1 on drone-cache-s3mock-67d655499f-gxr29 with │
│ 2022-09-09 18:31:00.479 INFO 1 --- [ main] c.a.testing.s3mock.S3MockApplication : No active profile set, falling back to 1 default profile: "default" ││ 2022-09-09 18:31:01.281 INFO 1 --- [ main] org.eclipse.jetty.util.log : Logging initialized @1574ms to org.eclipse.jetty.util.log.Slf4jLog │
│ 2022-09-09 18:31:01.393 INFO 1 --- [ main] o.s.b.w.e.j.JettyServletWebServerFactory : Server initialized with port: 9191 ││ 2022-09-09 18:31:01.408 INFO 1 --- [ main] org.eclipse.jetty.server.Server : jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d04 │
│ 2022-09-09 18:31:01.437 INFO 1 --- [ main] o.e.j.s.h.ContextHandler.application : Initializing Spring embedded WebApplicationContext ││ 2022-09-09 18:31:01.437 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 918 ms │
│ 2022-09-09 18:31:01.476 WARN 1 --- [ main] i.m.c.i.binder.jvm.JvmGcMetrics : GC notifications will not be available because com.sun.management.GarbageCollectionNotific ││ 2022-09-09 18:31:01.753 INFO 1 --- [ main] org.eclipse.jetty.server.session : DefaultSessionIdManager workerName=node0 │
│ 2022-09-09 18:31:01.753 INFO 1 --- [ main] org.eclipse.jetty.server.session : No SessionScavenger set, using defaults ││ 2022-09-09 18:31:01.755 INFO 1 --- [ main] org.eclipse.jetty.server.session : node0 Scavenging every 600000ms │
│ 2022-09-09 18:31:01.759 INFO 1 --- [ main] o.e.jetty.server.handler.ContextHandler : Started o.s.b.w.e.j.JettyEmbeddedWebAppContext@6273c5a4{application,/,[file:///tmp/jetty-d ││ 2022-09-09 18:31:01.760 INFO 1 --- [ main] org.eclipse.jetty.server.Server : Started @2054ms │
│ 2022-09-09 18:31:01.799 INFO 1 --- [ main] c.a.t.s3mock.store.StoreConfiguration : Using existing folder "/s3mockroot" as root folder. Will retain files on exit: true ││ 2022-09-09 18:31:01.800 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.spri │
│ 2022-09-09 18:31:01.803 INFO 1 --- [ main] org.eclipse.jetty.server.session : node0 Stopped scavenging ││ 2022-09-09 18:31:01.804 INFO 1 --- [ main] o.e.jetty.server.handler.ContextHandler : Stopped o.s.b.w.e.j.JettyEmbeddedWebAppContext@6273c5a4{application,/,[file:///tmp/jetty-d │
│ 2022-09-09 18:31:01.814 INFO 1 --- [ main] ConditionEvaluationReportLoggingListener : ││ │
│ Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. ││ 2022-09-09 18:31:01.830 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed │
│ ││ org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bucketStore' defined in class path resource [com/adobe/testing/s3mock/store/StoreConfiguration.clas │
│ at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) ~[spring-beans-5.3.22.jar!/:5.3.22] ││ at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638) ~[spring-beans-5.3.22.jar!/:5.3.22] │
│ at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.22.jar!/ ││ at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.22.jar!/:5.3.22] │
│ at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22] ││ at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22] │
│ at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22] ││ at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22] │
│ at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22] ││ at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22] │
│ at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) ~[spring-beans-5.3.22.jar!/:5.3.22] ││ at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.22.jar!/:5.3.22] │
│ at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.22.jar!/:5.3.22] ││ at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.3.jar!/:2.7.3] │
│ at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) ~[spring-boot-2.7.3.jar!/:2.7.3] ││ at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) ~[spring-boot-2.7.3.jar!/:2.7.3] │
│ at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.7.3.jar!/:2.7.3] ││ at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164) ~[spring-boot-2.7.3.jar!/:2.7.3] │
│ at com.adobe.testing.s3mock.S3MockApplication.start(S3MockApplication.java:181) ~[classes!/:na] ││ at com.adobe.testing.s3mock.S3MockApplication.start(S3MockApplication.java:145) ~[classes!/:na] │
│ at com.adobe.testing.s3mock.S3MockApplication.main(S3MockApplication.java:134) ~[classes!/:na] ││ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] │
│ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na] │
│ at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na] │
│ at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na] │
│ at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[s3mock.jar:na] │
│ at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[s3mock.jar:na] │
│ at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[s3mock.jar:na] │
│ at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[s3mock.jar:na] │
│ Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.adobe.testing.s3mock.store.BucketStore]: Factory method 'bucketStore' threw exception; nested e │
│ at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.3.22.jar!/:5.3.22] │
│ at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-5.3.22.jar!/:5.3.22] │
│ ... 28 common frames omitted │
│ Caused by: java.lang.NullPointerException: Cannot enter synchronized block because the return value of "java.util.Map.get(Object)" is null │
│ at com.adobe.testing.s3mock.store.BucketStore.getBucketMetadata(BucketStore.java:93) ~[classes!/:na] │
│ at com.adobe.testing.s3mock.store.BucketStore.createBucket(BucketStore.java:182) ~[classes!/:na] │
│ at com.adobe.testing.s3mock.store.BucketStore.lambda$new$0(BucketStore.java:64) ~[classes!/:na] │
│ at java.base/java.util.ArrayList.forEach(Unknown Source) ~[na:na] │
│ at com.adobe.testing.s3mock.store.BucketStore.<init>(BucketStore.java:64) ~[classes!/:na] │
│ at com.adobe.testing.s3mock.store.StoreConfiguration.bucketStore(StoreConfiguration.java:51) ~[classes!/:na] │
│ at com.adobe.testing.s3mock.store.StoreConfiguration$$EnhancerBySpringCGLIB$$8317884a.CGLIB$bucketStore$1(<generated>) ~[classes!/:na] │
│ at com.adobe.testing.s3mock.store.StoreConfiguration$$EnhancerBySpringCGLIB$$8317884a$$FastClassBySpringCGLIB$$2351a42b.invoke(<generated>) ~[classes!/:na] │
│ at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.3.22.jar!/:5.3.22] │
│ at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.3.22.jar!/:5.3.22] │
│ at com.adobe.testing.s3mock.store.StoreConfiguration$$EnhancerBySpringCGLIB$$8317884a.bucketStore(<generated>) ~[classes!/:na] │
│ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] │
│ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na] │
│ at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na] │
│ at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na] │
│ at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.22.jar!/:5.3.22] │
│ ... 29 common frames omitted
hey I'm really looking forward to this fix - I'm trying to replace localstack with anything that offers some kind of data persistence so that I can just tear down containers with docker compose down and not worry about my local project data
Hi, I am also interested in this. The pull request mentioned above seems to fix the startup problem, if data already exists. Could some please merge and update docker hub? Thanks in advance.
unfortunately, it's not as simple as just merging the PR - this still does not support recovering all data, also there is no integration test yet as this is very hard to test.
I just released 2.9.0 which lets you start S3Mock on an existing S3Mock root, if it was created with 2.9.0