conductor icon indicating copy to clipboard operation
conductor copied to clipboard

Published 20 hours ago • verified publisher icon Netflix

`docker-compose up` throws redis exception at startup

Open dragorosson opened this issue 1 year ago • 26 comments

Describe the bug

This is a follow-on to #3534. That specific issue is resolved by #3535, but further problems remain. When conductor-server starts up now with redis, which was added in #3435, it throws an exception:

18435 [main] INFO  com.netflix.conductor.redis.config.RedisSentinelConfiguration [] - Starting conductor server using redis_standalone.
2023-03-15 15:45:40 38812 [main] ERROR com.netflix.conductor.redis.dao.RedisMetadataDAO [] - refresh TaskDefs failed 
2023-03-15 15:45:40 redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
2023-03-15 15:45:40     at redis.clients.jedis.util.Pool.getResource(Pool.java:84) ~[jedis-3.8.0.jar!/:?]
2023-03-15 15:45:40     at redis.clients.jedis.JedisPool.getResource(JedisPool.java:377) ~[jedis-3.8.0.jar!/:?]
2023-03-15 15:45:40     at com.netflix.conductor.redis.jedis.JedisStandalone.executeInJedis(JedisStandalone.java:53) ~[conductor-redis-persistence-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
2023-03-15 15:45:40     at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]

Details Conductor version: 3f81d3f12a305754578150d4bd15f017f9a889fe (current main) Persistence implementation: whatever comes with the default docker-compose up now Queue implementation: RabbitMQ I guess Lock: Redis? Workflow definition: N/A Task definition: N/A Event handler definition: N/A

To Reproduce Steps to reproduce the behavior:

  1. Use the docker-compose up tutorial at https://conductor.netflix.com/devguide/running/docker.html

dragorosson avatar Mar 15 '23 21:03 dragorosson

I'll add to this. I was able to recreate this as well. I had to remove the health check for RabbitMQ because Docker will not see that it is healthy in the current state because it hangs up in a weird state. Once you remove the health check from conductor-server everything comes up. Here is the entire stack trace from conductor-server.

conductor-server    | 136932 [pool-18-thread-1] ERROR com.netflix.conductor.core.execution.tasks.SystemTaskWorker [] - Error polling system task in queue:SUB_WORKFLOW
conductor-server    | java.lang.RuntimeException: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
conductor-server    | 	at io.orkes.conductor.mq.redis.single.RedisQueueMonitor.loadScript(RedisQueueMonitor.java:81) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server    | 	at io.orkes.conductor.mq.redis.single.RedisQueueMonitor.<init>(RedisQueueMonitor.java:39) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server    | 	at io.orkes.conductor.mq.redis.single.ConductorRedisQueue.<init>(ConductorRedisQueue.java:50) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server    | 	at io.orkes.conductor.queue.dao.RedisQueueDAO.getConductorQueue(RedisQueueDAO.java:47) ~[orkes-conductor-queues-1.0.3.jar!/:?]
conductor-server    | 	at io.orkes.conductor.queue.dao.BaseRedisQueueDAO.lambda$get$0(BaseRedisQueueDAO.java:57) ~[orkes-conductor-queues-1.0.3.jar!/:?]
conductor-server    | 	at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source) ~[?:?]
conductor-server    | 	at io.orkes.conductor.queue.dao.BaseRedisQueueDAO.get(BaseRedisQueueDAO.java:57) ~[orkes-conductor-queues-1.0.3.jar!/:?]
conductor-server    | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
conductor-server    | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
conductor-server    | 	at java.lang.Thread.run(Unknown Source) ~[?:?]
conductor-server    | Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
conductor-server    | 	at redis.clients.jedis.util.Pool.getResource(Pool.java:84) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:377) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:15) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
conductor-server    | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
conductor-server    | 	at java.lang.Thread.run(Unknown Source) ~[?:?]
conductor-server    | Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
conductor-server    | 	at redis.clients.jedis.util.Pool.getResource(Pool.java:84) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:377) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:15) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at io.orkes.conductor.mq.redis.single.RedisQueueMonitor.loadScript(RedisQueueMonitor.java:75) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server    | 	... 15 more
conductor-server    | Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed to create socket.
conductor-server    | 	at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:110) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at redis.clients.jedis.Connection.connect(Connection.java:226) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:144) ~[jedis-3.8.0.jar!/:?]
conductor-server    | 	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:314) ~[jedis-3.8.0.jar!/:?]

bdparrish avatar Mar 18 '23 21:03 bdparrish

I tried running the Conductor locally too and got the same problems: with Redis and with RabbitMQ.

Talking about Redis I think there are 2 issues:

  1. port misconfiguration in config-local.properties
  2. despite fixing the above I was still having problems while containers were using internal network & trying to resolve hostname rs. I've removed it (switched to default network) and everything seems to work fine now

Above changes made on a fork: https://github.com/Netflix/conductor/commit/20eab6dbe465799bfa63d92ce62904ec38a3a01d

karczewsky avatar Mar 19 '23 13:03 karczewsky

Good points @karczewsky - I was able to add redis to the internal network and it worked as expected. However now is the next issue...

On startup es is failing -

3665 [main] ERROR com.netflix.conductor.es6.dao.index.ElasticSearchRestDAOV6 [] - Failed to init template_task_log
conductor-server    | org.elasticsearch.client.ResponseException: method [PUT], host [http://es:9200], URI [/_template/template_task_log], status line [HTTP/1.1 400 Bad Request]
conductor-server    | Warnings: [Legacy index templates are deprecated in favor of composable templates.]
conductor-server    | {"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [template] in the template "}],"type":"parse_exception","reason":"unknown key [template] in the template "},"status":400}
conductor-server    | 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936) ~[elasticsearch-rest-client-6.8.12.jar!/:6.8.12]
conductor-server    | 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.12.jar!/:6.8.12]
conductor-server    | 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:327) ~[elasticsearch-rest-client-6.8.12.jar!/:6.8.12]
conductor-server    | 	at com.netflix.conductor.es6.dao.index.ElasticSearchRestDAOV6.initIndexTemplate(ElasticSearchRestDAOV6.java:274) ~[conductor-es6-persistence-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]
conductor-server    | 	at com.netflix.conductor.es6.dao.index.ElasticSearchRestDAOV6.initIndexesTemplates(ElasticSearchRestDAOV6.java:257) ~[conductor-es6-persistence-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]
conductor-server    | 	at com.netflix.conductor.es6.dao.index.ElasticSearchRestDAOV6.createIndexesTemplates(ElasticSearchRestDAOV6.java:247) ~[conductor-es6-persistence-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]
conductor-server    | 	at com.netflix.conductor.es6.dao.index.ElasticSearchRestDAOV6.setup(ElasticSearchRestDAOV6.java:239) ~[conductor-es6-persistence-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]
conductor-server    | 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
conductor-server    | 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
conductor-server    | 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
conductor-server    | 	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
conductor-server    | 	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1609) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1573) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1462) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1349) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) ~[spring-beans-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.22.jar!/:5.3.22]
conductor-server    | 	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.3.jar!/:2.7.3]
conductor-server    | 	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) ~[spring-boot-2.7.3.jar!/:2.7.3]
conductor-server    | 	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) ~[spring-boot-2.7.3.jar!/:2.7.3]
conductor-server    | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.7.3.jar!/:2.7.3]
conductor-server    | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.7.3.jar!/:2.7.3]
conductor-server    | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-2.7.3.jar!/:2.7.3]
conductor-server    | 	at com.netflix.conductor.Conductor.main(Conductor.java:39) ~[classes!/:3.14.0-SNAPSHOT]
conductor-server    | 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
conductor-server    | 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
conductor-server    | 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
conductor-server    | 	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
conductor-server    | 	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[conductor-server-3.14.0-SNAPSHOT-boot.jar:3.14.0-SNAPSHOT]
conductor-server    | 	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[conductor-server-3.14.0-SNAPSHOT-boot.jar:3.14.0-SNAPSHOT]
conductor-server    | 	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[conductor-server-3.14.0-SNAPSHOT-boot.jar:3.14.0-SNAPSHOT]
conductor-server    | 	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[conductor-server-3.14.0-SNAPSHOT-boot.jar:3.14.0-SNAPSHOT]
conductor-server    | Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [http://es:9200], URI [/_template/template_task_log], status line [HTTP/1.1 400 Bad Request]
conductor-server    | Warnings: [Legacy index templates are deprecated in favor of composable templates.]
conductor-server    | {"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [template] in the template "}],"type":"parse_exception","reason":"unknown key [template] in the template "},"status":400}
conductor-server    | 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:552) ~[elasticsearch-rest-client-6.8.12.jar!/:6.8.12]
conductor-server    | 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537) ~[elasticsearch-rest-client-6.8.12.jar!/:6.8.12]
conductor-server    | 	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) ~[httpcore-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182) ~[httpasyncclient-4.1.5.jar!/:4.1.5]
conductor-server    | 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) ~[httpasyncclient-4.1.5.jar!/:4.1.5]
conductor-server    | 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) ~[httpasyncclient-4.1.5.jar!/:4.1.5]
conductor-server    | 	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.15.jar!/:4.4.15]
conductor-server    | 	at java.lang.Thread.run(Unknown Source) ~[?:?]

bdparrish avatar Mar 19 '23 15:03 bdparrish

Heads Up: I'm pretty new to the Conductor.

I see that recently there was a change in docker-compose from elastic 6.x.x to 8.x.x. Maybe that's a clue - could be some breaking API change. The code seems to require either es6 or es7.

karczewsky avatar Mar 19 '23 17:03 karczewsky

I am new as well.

Yes, this is all from the recent update. Docker works as expected with the previous configuration.

I want to get the updated changes because it looks like some items are in pace to make SQS connections easier.

bdparrish avatar Mar 19 '23 17:03 bdparrish

Good points @karczewsky - I was able to add redis to the internal network and it worked as expected. However now is the next issue...

On startup es is failing -

The elasticsearch image bump to 8.6.2 was in #3503. I was seeing the es failure too. I wanted to create an issue for it, but when I tried to reproduce it again by just reverting #3435 (leaving #3503) I wasn't seeing the exception. I don't know what would make that difference, but I wanted to throw that out there.

dragorosson avatar Mar 19 '23 18:03 dragorosson

I didn't see anything in the diff between the two that stands out, but this is killing my analysis to use Netflix Conductor at all.

bdparrish avatar Mar 19 '23 23:03 bdparrish

@bdparrish we are going to take a look.

v1r3n avatar Mar 20 '23 18:03 v1r3n

I encountered a similar problem:

conductor-server  | java.lang.RuntimeException: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
conductor-server  | 	at io.orkes.conductor.mq.redis.single.RedisQueueMonitor.loadScript(RedisQueueMonitor.java:81) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server  | 	at io.orkes.conductor.mq.redis.single.RedisQueueMonitor.<init>(RedisQueueMonitor.java:39) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server  | 	at io.orkes.conductor.mq.redis.single.ConductorRedisQueue.<init>(ConductorRedisQueue.java:50) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server  | 	at io.orkes.conductor.queue.dao.RedisQueueDAO.getConductorQueue(RedisQueueDAO.java:47) ~[orkes-conductor-queues-1.0.3.jar!/:?]
conductor-server  | 	at io.orkes.conductor.queue.dao.BaseRedisQueueDAO.lambda$get$0(BaseRedisQueueDAO.java:57) ~[orkes-conductor-queues-1.0.3.jar!/:?]
conductor-server  | 	at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source) ~[?:?]
conductor-server  | 	at io.orkes.conductor.queue.dao.BaseRedisQueueDAO.get(BaseRedisQueueDAO.java:57) ~[orkes-conductor-queues-1.0.3.jar!/:?]
conductor-server  | 	at io.orkes.conductor.queue.dao.BaseRedisQueueDAO.pop(BaseRedisQueueDAO.java:108) ~[orkes-conductor-queues-1.0.3.jar!/:?]
conductor-server  | 	at com.netflix.conductor.core.execution.tasks.SystemTaskWorker.pollAndExecute(SystemTaskWorker.java:111) ~[conductor-core-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]
conductor-server  | 	at com.netflix.conductor.core.execution.tasks.SystemTaskWorker.lambda$startPolling$0(SystemTaskWorker.java:81) ~[conductor-core-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]
conductor-server  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
conductor-server  | 	at java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[?:?]
conductor-server  | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
conductor-server  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
conductor-server  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
conductor-server  | 	at java.lang.Thread.run(Unknown Source) ~[?:?]
conductor-server  | Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
conductor-server  | 	at redis.clients.jedis.util.Pool.getResource(Pool.java:84) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:377) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:15) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at io.orkes.conductor.mq.redis.single.RedisQueueMonitor.loadScript(RedisQueueMonitor.java:75) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server  | 	... 15 more
conductor-server  | Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed to create socket.
conductor-server  | 	at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:110) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.Connection.connect(Connection.java:226) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:144) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:314) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:92) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:297) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.Jedis.<init>(Jedis.java:169) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:177) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571) ~[commons-pool2-2.11.1.jar!/:2.11.1]
conductor-server  | 	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:298) ~[commons-pool2-2.11.1.jar!/:2.11.1]
conductor-server  | 	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223) ~[commons-pool2-2.11.1.jar!/:2.11.1]
conductor-server  | 	at redis.clients.jedis.util.Pool.getResource(Pool.java:75) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:377) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:15) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at io.orkes.conductor.mq.redis.single.RedisQueueMonitor.loadScript(RedisQueueMonitor.java:75) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server  | 	... 15 more
conductor-server  | Caused by: java.net.UnknownHostException: rs
conductor-server  | 	at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?]
conductor-server  | 	at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
conductor-server  | 	at java.net.Socket.connect(Unknown Source) ~[?:?]
conductor-server  | 	at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:80) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.Connection.connect(Connection.java:226) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:144) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:314) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:92) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:297) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.Jedis.<init>(Jedis.java:169) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:177) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571) ~[commons-pool2-2.11.1.jar!/:2.11.1]
conductor-server  | 	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:298) ~[commons-pool2-2.11.1.jar!/:2.11.1]
conductor-server  | 	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223) ~[commons-pool2-2.11.1.jar!/:2.11.1]
conductor-server  | 	at redis.clients.jedis.util.Pool.getResource(Pool.java:75) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:377) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:15) ~[jedis-3.8.0.jar!/:?]
conductor-server  | 	at io.orkes.conductor.mq.redis.single.RedisQueueMonitor.loadScript(RedisQueueMonitor.java:75) ~[orkes-redis-queues-1.0.3.jar!/:?]
conductor-server  | 	... 15 more

Is this same to yours? And is this solved now? It really messed me up.

darcyzhc avatar Mar 24 '23 17:03 darcyzhc

hello,

Any news about this issue and how to fix it ? Currently, I use conductor with postgres and no problem with the version 3.13.3 but with the new version (tag 3.13.5), I have many logs as @darcyzhc

Graynder avatar Mar 30 '23 08:03 Graynder

I am having the same issue right off the bat when running docker compose up. I am using WSL 2.0 and getting failure on Rabbitmq. Anyone able to figure it out!

dependency failed to start: container for service "rabbitmq" exited (0)

akramh avatar Apr 04 '23 04:04 akramh

I just noticed https://github.com/Netflix/conductor/pull/3543 which was opened two weeks ago, approved, but still not merged. @v1r3n @manan164 what's the status of the PR and does it fix this issue?

dragorosson avatar Apr 04 '23 15:04 dragorosson

@Graynder I seem to be facing issue with connecting to postgres, can you help me with what changes you did to connect to postgres?

coolabhi95 avatar Apr 05 '23 07:04 coolabhi95

@darcyzhc were you able to resolve the issue?

coolabhi95 avatar Apr 05 '23 07:04 coolabhi95

I'm getting the same issue. Does anyone resolve it?

RomanAlberdaSoftserveInc avatar Jun 06 '23 11:06 RomanAlberdaSoftserveInc

no fix discover so I use old version off conductor because I don't want to install a redis for the moment ^^'

Graynder avatar Jun 08 '23 12:06 Graynder

i'm also stuck with the same issue, i tried using postgres and dynomite but both doesn't work. pls suggest old stable version to proceed where postgres and dynomite docker works

srichandar avatar Jun 20 '23 12:06 srichandar

same... stuck on this issue.

JoshCrosby avatar Jun 27 '23 19:06 JoshCrosby

any updates on this?

riemann89 avatar Jul 08 '23 21:07 riemann89

I am hitting same issue when trying to execute with postgres yaml. I am struck with this issue. please provide a solution for this. thanks

missioncoding avatar Jul 10 '23 16:07 missioncoding

Hi @RomanAlberdaSoftserveInc , @Graynder , @srichandar , @JoshCrosby @riemann89 and @missioncoding

Can you please share the configuration you are using? Please mask the sensitive fields. This PR fixes the redis issue. We can chat here for more realtime collaboration.

manan164 avatar Jul 10 '23 16:07 manan164

Hello @manan164 thanks for the response. I didn't make any configuration change apart from adding the conductor.postgres.persistance runtime dependency in the server/build.gradle which solves another issue of MetaDataDao constructor parameter.

After which i applied docker compose -f docker-compose.yaml -f docker-compose-postgres.yaml build command followed with up

the exceptions traces going infinitely

no changes apart from these steps.

missioncoding avatar Jul 11 '23 04:07 missioncoding

Hi all, I have raised PR for the fix,https://github.com/Netflix/conductor/pull/3692 Please pull the changes manually if this is blocking the use case.

manan164 avatar Jul 11 '23 14:07 manan164

Folks, Just a note on this, If you are still seeing the issue please run docker compose build before running docker compose up since the docker image needs to pick the property file changes done in the #3692

manan164 avatar Jul 13 '23 17:07 manan164

Hi @manan164 , from what I can see in the PR, a Redis service is added, but that's precisely the problem. With version 3.13.3 (the version I'm currently using), there's no need for Redis, but with version 3.13.5, I have the impression that a Redis service is required for it to work, and this is a change of breacking from my point of view, which is blocking migration to a higher version.

Graynder avatar Jul 25 '23 12:07 Graynder

Hi @Graynder , Since we are now using orkes-queue, redis is required. Do you see any issue in 3.13.3 ?

manan164 avatar Jul 26 '23 07:07 manan164