apache
[Bug] Error with S3 Configuration During Helm Deployment of DolphinScheduler
Search before asking
- [X] I had searched in the issues and found no similar issues.
What happened
I encountered an issue while deploying DolphinScheduler using Helm. The deployment fails with an error related to S3 configuration. Below are the details of the error and my setup.
What you expected to happen
Error log:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageOperate' defined in org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.plugin.storage.api.StorageOperator]: Factory method 'storageOperate' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) at org.apache.dolphinscheduler.server.master.MasterServer.main(MasterServer.java:102) Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.plugin.storage.api.StorageOperator]: Factory method 'storageOperate' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ... 19 common frames omitted Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:4062) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1278) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1268) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1406) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperator.exceptionWhenBucketNameNotExists(S3StorageOperator.java:209) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperator.<init>(S3StorageOperator.java:71) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperatorFactory.createStorageOperate(S3StorageOperatorFactory.java:34) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.lambda$storageOperate$0(StorageConfiguration.java:43) at java.util.Optional.map(Optional.java:215) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.storageOperate(StorageConfiguration.java:38) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190.CGLIB$storageOperate$0(<generated>) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190$$FastClassBySpringCGLIB$$d3a2ba71.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190.storageOperate(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 20 common frames omitted Caused by: java.net.UnknownHostException: s3 at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764) at java.net.InetAddress.getAllByName0(InetAddress.java:1291) at java.net.InetAddress.getAllByName(InetAddress.java:1144) at java.net.InetAddress.getAllByName(InetAddress.java:1065) at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy191.connect(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ... 49 common frames omitted [WI-0][TI-0] - 2024-12-04 00:02:55.317 ERROR [Master-Server] o.a.d.c.t.DefaultUncaughtExceptionHandler:[44] - Caught an exception in Thread[Master-Server,5,main]. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageOperate' defined in org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.plugin.storage.api.StorageOperator]: Factory method 'storageOperate' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) at org.apache.dolphinscheduler.server.master.MasterServer.main(MasterServer.java:102) Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.plugin.storage.api.StorageOperator]: Factory method 'storageOperate' threw exception; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ... 19 common frames omitted Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: s3 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:4062) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1278) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1268) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1406) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperator.exceptionWhenBucketNameNotExists(S3StorageOperator.java:209) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperator.<init>(S3StorageOperator.java:71) at org.apache.dolphinscheduler.plugin.storage.s3.S3StorageOperatorFactory.createStorageOperate(S3StorageOperatorFactory.java:34) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.lambda$storageOperate$0(StorageConfiguration.java:43) at java.util.Optional.map(Optional.java:215) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.storageOperate(StorageConfiguration.java:38) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190.CGLIB$storageOperate$0(<generated>) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190$$FastClassBySpringCGLIB$$d3a2ba71.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$33e8a190.storageOperate(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 20 common frames omitted Caused by: java.net.UnknownHostException: s3 at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764) at java.net.InetAddress.getAllByName0(InetAddress.java:1291) at java.net.InetAddress.getAllByName(InetAddress.java:1144) at java.net.InetAddress.getAllByName(InetAddress.java:1065) at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy191.connect(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ... 49 common frames omitted
Helm Values Configuration:
conf: auto: false common: data.basedir.path: /tmp/dolphinscheduler resource.storage.type: S3 resource.storage.upload.base.path: /dolphinscheduler resource.aws.access.key.id: id resource.aws.secret.access.key: key resource.aws.region: us-east-1 resource.aws.s3.bucket.name: emr-cdcicd-test resource.aws.s3.endpoint: https://S3.us-east-1.amazonaws.com resource.hdfs.fs.defaultFS: s3a://dolphinscheduler
I make sure s3 key is avaiable.
And This is my step to install dolphinscheduler.
export VERSION=3.2.2
helm pull oci://registry-1.docker.io/apache/dolphinscheduler-helm --version ${VERSION}
tar -xvf dolphinscheduler-helm-${VERSION}.tgz
cd dolphinscheduler-helm
helm repo add bitnami https://charts.bitnami.com/bitnami
helm dependency update .
helm install dolphinscheduler . -n dolphin
How to reproduce
export VERSION=3.2.2 helm pull oci://registry-1.docker.io/apache/dolphinscheduler-helm --version ${VERSION} tar -xvf dolphinscheduler-helm-${VERSION}.tgz cd dolphinscheduler-helm helm repo add bitnami https://charts.bitnami.com/bitnami helm dependency update . helm install dolphinscheduler . -n dolphin
Anything else
No response
Version
3.2.x
Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
