spring-cloud-openfeign Feign client lazy initialization fails when using CompletableFuture.supplyAsync() to call the Feign client initially

Describe the bug Feign client initialization is always lazy. As such, the first calling thread triggers the initialization, which may be another Spring component (e.g. REST controller) using CompletableFuture.supplyAsync() to do so. This uses ForkJoinWorkerThread underneath, resulting in a ClassNotFoundException, if and only if running in a dockerized environment based on Spring Boot Build Image plugin.

Using other JVM Executors(except newWorkStealingExecutor), e.g. fixedSizeThreadExecutor(), does not result in this behavior. Also eagerly initializing the Feign client by calling it synchronously, e.g. from within a CommandLineRunner, circumvents this. Once, the Feign client is properly initialized, there is no problem, however, Feign does not yet support eager initialization.

Sample A sample project with instructions to reproduce is provided here: https://github.com/maverick1601/openfeign-build-image-error

Jan 28 '21 12:01 maverick1601

I was able to reproduce it with the sample. Example stacktrace:

 java.lang.ClassNotFoundException: org.springframework.boot.autoconfigure.condition.OnPropertyCondition
      at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[na:na]
      at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[na:na]
      at java.base/java.lang.ClassLoader.loadClass(Unknown Source) ~[na:na]
      at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
      at java.base/java.lang.Class.forName(Unknown Source) ~[na:na]
      at org.springframework.util.ClassUtils.forName(ClassUtils.java:284) ~[spring-core-5.3.3.jar:5.3.3]
      at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:324) ~[spring-core-5.3.3.jar:5.3.3]
      at org.springframework.context.annotation.ConditionEvaluator.getCondition(ConditionEvaluator.java:124) ~[spring-context-5.3.3.jar:5.3.3]
      at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:96) ~[spring-context-5.3.3.jar:5.3.3]
      at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:88) ~[spring-context-5.3.3.jar:5.3.3]
      at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:71) ~[spring-context-5.3.3.jar:5.3.3]
       at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.doRegisterBean(AnnotatedBeanDefinitionReader.java:254) ~[spring-context-5.3.3.jar:5.3.3]
       at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.registerBean(AnnotatedBeanDefinitionReader.java:147) ~[spring-context-5.3.3.jar:5.3.3]
       at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.register(AnnotatedBeanDefinitionReader.java:137) ~[spring-context-5.3.3.jar:5.3.3]
      at org.springframework.context.annotation.AnnotationConfigApplicationContext.register(AnnotationConfigApplicationContext.java:168) ~[spring-context-5.3.3.jar:5.3.3]
      at org.springframework.cloud.context.named.NamedContextFactory.createContext(NamedContextFactory.java:122) ~[spring-cloud-context-3.0.1.jar:3.0.1]
      at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:101) ~[spring-cloud-context-3.0.1.jar:3.0.1]
       at org.springframework.cloud.context.named.NamedContextFactory.getInstances(NamedContextFactory.java:181) ~[spring-cloud-context-3.0.1.jar:3.0.1]
       at org.springframework.cloud.openfeign.loadbalancer.FeignBlockingLoadBalancerClient.execute(FeignBlockingLoadBalancerClient.java:85) ~[spring-cloud-openfeign-core-3.0.1.jar:3.0.1]
      at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:119) ~[feign-core-10.10.1.jar:na]
       at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89) ~[feign-core-10.10.1.jar:na]
       at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100) ~[feign-core-10.10.1.jar:na]
       at com.sun.proxy.$Proxy73.fetchIt(Unknown Source) ~[na:na]
       at de.digitalfrontiers.spring.build.image.SampleController$sampleApi$1.get(SampleController.kt:14) ~[classes/:na]
       at de.digitalfrontiers.spring.build.image.SampleController$sampleApi$1.get(SampleController.kt:9) ~[classes/:na]
      at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[na:na]
       at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(Unknown Source) ~[na:na]
       at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[na:na]
       at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) ~[na:na]
       at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source) ~[na:na]
       at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) ~[na:na]
       at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) ~[na:na]

Jan 29 '21 14:01 OlgaMaciaszek

@maverick1601, thanks for reporting this. Will discuss it with the team and get back to you.

Jan 29 '21 14:01 OlgaMaciaszek

@maverick1601 we don't recommend using CompletableFuture.supplyAsync(), however, an enhancement that could be helpful here would be to offer an opt-in eager-loading mechanism. We don't think it's a priority at this point, so we would not work on this right now, however, we would welcome pull requests. As a reference, here is a PR, that introduced eager-loading in SC Ribbon where child contexts were being lazily initialised, very similarly to what we have in OpenFeign: https://github.com/spring-cloud/spring-cloud-netflix/commit/567a6c4a9f3a39ecfeb76d80f33c585c2ca1695a.

Feb 18 '21 16:02 OlgaMaciaszek

Hi, is there any news about this issue and the associated PR? I encountered a similar issue when using openfeign client called within a method annotated as @Async that uses a ForkJoinPool executor (spring-boot: 2.5.5, spring-cloud: 2020.0.4). I have yet to find a solution or workaround.

P.S. I think it's related to #600

P.S.S. I've tried the solution described in #600 and I managed to make it working. I hope that it can be fixed asap because it's not only a openfeign issue but it does apply on LoadBalanced RestTemplates too

Oct 04 '21 16:10 davgia

This issue is not within the team's priorities right now, but is open to community contributions.

Oct 28 '21 16:10 OlgaMaciaszek

An alternative solution is to use rxjava,

mvn dep

io.reactivex rxjava

Mar 01 '22 17:03 mgrigis

The previous workaround didn't work for me as the constructor of LoadBalancerClientFactory without arguments is deprecated, as a result that solution causes NPE.

Here is the modified version:

    @Bean
    @ConditionalOnMissingBean
    public LoadBalancerClientFactory loadBalancerClientFactory(LoadBalancerClientsProperties properties) {
        return new LoadBalancerClientFactory(properties) {
            @Override
            protected AnnotationConfigApplicationContext createContext(String name) {
                // FIXME: temporary switch classloader to use the correct one when creating the context
                ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
                Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());
                AnnotationConfigApplicationContext context = super.createContext(name);
                Thread.currentThread().setContextClassLoader(originalClassLoader);
                return context;
            }
        };
    }

Apr 28 '22 07:04 DenisKorolev

The previous workaround didn't work for me as the constructor of LoadBalancerClientFactory without arguments is deprecated, as a result that solution causes NPE.

@DenisKorolev is there any particular reason why you have omitted the following line from the previous workaround?

clientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));

This line still appears in the org.springframework.cloud.loadbalancer.config.LoadBalancerAutoConfiguration.

May 10 '22 06:05 timosmit

I had the same problem

Jun 17 '22 02:06 luoyi2019

jdk11 had the problem, but jdk8 was ok.

Jul 01 '22 08:07 lgbwonder

I just encountered the same issue when the LoadBalancerClient is initialized from within an R2DBC transaction (my service is @Transactional and performs load-balanced WebClient requests with DeferringLoadBalancerExchangeFilterFunction as automatically configured when using a @LoadBalanced WebClient.Builder).

I'm using the Oracle R2DBC Driver which performs its work on ForkJoinPool.common. Fortunately, I was able to supply my own Executor to work around this issue.

Aug 12 '22 09:08 kzander91

I'm getting this OnBeanCondition error. Seems quite similar. As soon as my CompletableFuture<Optional<Object>> starts to execute, it fails with below logs. I can't figure out what's the issue here. On local dev machine it's running good, fails on ec2 linux.

java.lang.IllegalStateException: Failed to execute CommandLineRunner
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:779) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:322) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
        at com.myapp.cricket.app.Application.main(Application.java:40) ~[classes!/:1.0.0-SNAPSHOT.1]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[app.jar:1.0.0-SNAPSHOT.1]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:109) ~[app.jar:1.0.0-SNAPSHOT.1]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[app.jar:1.0.0-SNAPSHOT.1]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[app.jar:1.0.0-SNAPSHOT.1]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Could not find class [org.springframework.boot.autoconfigure.condition.OnBeanCondition]
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[na:na]
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2063) ~[na:na]
        at com.myapp.cricket.app.bootstrap.SpmConfigLookupRunner.run(SpmConfigLookupRunner.java:72) ~[classes!/:1.0.0-SNAPSHOT.1]
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795) ~[spring-boot-2.3.2.RELEASE.jar!/:2.3.2.RELEASE]
        ... 11 common frames omitted
Caused by: java.lang.IllegalArgumentException: Could not find class [org.springframework.boot.autoconfigure.condition.OnBeanCondition]
        at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:334) ~[spring-core-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.context.annotation.ConditionEvaluator.getCondition(ConditionEvaluator.java:124) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:96) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:88) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:71) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.doRegisterBean(AnnotatedBeanDefinitionReader.java:254) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.registerBean(AnnotatedBeanDefinitionReader.java:147) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.register(AnnotatedBeanDefinitionReader.java:137) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.context.annotation.AnnotationConfigApplicationContext.register(AnnotationConfigApplicationContext.java:162) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.cloud.context.named.NamedContextFactory.createContext(NamedContextFactory.java:120) ~[spring-cloud-context-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:102) ~[spring-cloud-context-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.cloud.context.named.NamedContextFactory.getInstance(NamedContextFactory.java:146) ~[spring-cloud-context-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.cloud.loadbalancer.support.LoadBalancerClientFactory.getInstance(LoadBalancerClientFactory.java:60) ~[spring-cloud-loadbalancer-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.cloud.loadbalancer.blocking.client.BlockingLoadBalancerClient.choose(BlockingLoadBalancerClient.java:81) ~[spring-cloud-loadbalancer-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.cloud.client.loadbalancer.InterceptorRetryPolicy.canRetry(InterceptorRetryPolicy.java:59) ~[spring-cloud-commons-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.retry.support.RetryTemplate.canRetry(RetryTemplate.java:375) ~[spring-retry-1.2.5.RELEASE.jar!/:na]
        at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:278) ~[spring-retry-1.2.5.RELEASE.jar!/:na]
        at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:180) ~[spring-retry-1.2.5.RELEASE.jar!/:na]
        at org.springframework.cloud.client.loadbalancer.RetryLoadBalancerInterceptor.intercept(RetryLoadBalancerInterceptor.java:77) ~[spring-cloud-commons-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:77) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:739) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:612) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at com.myapp.cricket.app.web.exchange.ExchangeService.exchange(ExchangeService.java:85) ~[classes!/:1.0.0-SNAPSHOT.1]
        at com.myapp.cricket.app.web.exchange.ExchangeService$$FastClassBySpringCGLIB$$df1cc642.invoke(<generated>) ~[classes!/:1.0.0-SNAPSHOT.1]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) ~[spring-aop-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at com.myapp.cricket.app.web.exchange.ExchangeService$$EnhancerBySpringCGLIB$$eba28ab1.exchange(<generated>) ~[classes!/:1.0.0-SNAPSHOT.1]
        at com.myapp.cricket.app.service.SpmContinentAndCountryConfigLookupService.getContinentAndCountryByContinentId(SpmContinentAndCountryConfigLookupService.java:157) ~[classes!/:1.0.0-SNAPSHOT.1]
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[na:na]
        at com.myapp.cricket.app.service.SpmContinentAndCountryConfigLookupService.spmContinentAndCountryFetchUpdate(SpmContinentAndCountryConfigLookupService.java:125) ~[classes!/:1.0.0-SNAPSHOT.1]
        at com.myapp.cricket.app.service.SpmContinentAndCountryConfigLookupService.spmContinentAndCountryFetchUpdate(SpmContinentAndCountryConfigLookupService.java:67) ~[classes!/:1.0.0-SNAPSHOT.1]
        at com.myapp.cricket.app.bootstrap.SpmConfigLookupRunner.lambda$fetchSpmContinentsAndCountriesFuture$0(SpmConfigLookupRunner.java:84) ~[classes!/:1.0.0-SNAPSHOT.1]
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1764) ~[na:na]
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1756) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[na:na]
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.autoconfigure.condition.OnBeanCondition
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606) ~[na:na]
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168) ~[na:na]
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
        at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
        at java.base/java.lang.Class.forName(Class.java:468) ~[na:na]
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:284) ~[spring-core-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:324) ~[spring-core-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
        ... 43 common frames omitted

Aug 30 '22 12:08 ashutosh049

My problem like it, #1139

But my problem is deadlock, Maybe it's the same thing with different consequences

Found one Java-level deadlock:
=============================
"ForkJoinPool.commonPool-worker-2":
  waiting to lock monitor 0x000000002c456c18 (object 0x00000005c2284aa8, a java.util.concurrent.ConcurrentHashMap),
  which is held by "main"
"main":
  waiting to lock monitor 0x000000002bda1db8 (object 0x000000071649d0e0, a java.util.concurrent.ConcurrentHashMap),
  which is held by "ForkJoinPool.commonPool-worker-2"

Java stack information for the threads listed above:
===================================================
"ForkJoinPool.commonPool-worker-2":
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:179)
        - waiting to lock <0x00000005c2284aa8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:493)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:520)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:491)
        at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:227)
        at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:231)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1419)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1218)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1175)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostPro
cessor.java:595)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.
java:376)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1404)      
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:592)       
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$162/2081000371.getObject(Unknown Source)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        - locked <0x000000071a98c5a8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
        - locked <0x000000071a98c200> (a java.lang.Object)
        at org.springframework.cloud.context.named.NamedContextFactory.createContext(NamedContextFactory.java:136)
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:101)
        - locked <0x000000071649d0e0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getContext(SpringClientFactory.java:131)
        at org.springframework.cloud.context.named.NamedContextFactory.getInstance(NamedContextFactory.java:145)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getInstance(SpringClientFactory.java:121)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getClientConfig(SpringClientFactory.java:75)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.getClientConfig(LoadBalancerFeignClient.java:97)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:81)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:110)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
        at com.sun.proxy.$Proxy154.getPersonOne(Unknown Source)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig.lambda$initBean$0(TestDeadLockConfig.java:28)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$Lambda$565/2000449863.run(Unknown Source)
        at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1626)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java)
        at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1618)
        at java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"main":
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:100)
        - waiting to lock <0x000000071649d0e0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getContext(SpringClientFactory.java:131)
        at org.springframework.cloud.context.named.NamedContextFactory.getInstance(NamedContextFactory.java:145)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getInstance(SpringClientFactory.java:121)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getClientConfig(SpringClientFactory.java:75)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.getClientConfig(LoadBalancerFeignClient.java:97)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:81)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:110)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
        at com.sun.proxy.$Proxy153.getByAccountName(Unknown Source)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig.initBean(TestDeadLockConfig.java:33)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f.CGLIB$initBean$0(<generated>)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f$$FastClassBySpringCGLIB$$bc6c3884.invoke(<generated>)       
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)       
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f.initBean(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory
.java:1320)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1159)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)       
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$162/2081000371.getObject(Unknown Source)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        - locked <0x00000005c2284aa8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
        - locked <0x00000005c23cb290> (a java.lang.Object)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204)
        at com.siiri.cts.open.web.OpenServiceWebApplication.main(OpenServiceWebApplication.java:42)

Found 1 deadlock.

Sep 02 '22 02:09 DDAaTao

I'm facing the exact same problem after an update from Spring 2.3 to Spring 2.7 + Java update from 14 to 17. We are not using any @Async or CompletableFuture, but the feign client is called by a Grizzly Thread (started as part of a Spring Boot app) that is part of our core infrastructure and needs to call another running Spring Boot app via Feign. This used to work without any problems before the update. Disappointed way too often to see such an issue open for such a long time with no prospect of a solution

Oct 05 '22 06:10 HJK181

Made another sample project that reproduces a problem. It also contains a submodule with proposed workaround by @DenisKorolev. There are also a test with boot 2.3 and ribbon that works fine and a test for when eager-loading enabled for Spring's loadbalancer (available in boot 3+, but you can implement the same logic for yourself if it fits).

Oct 30 '22 00:10 Psimage

Hi. In what version was the "openfeign asynchronous call error "problem fixed?

Oct 31 '22 09:10 ruanun

This issue is still open.

Oct 31 '22 11:10 OlgaMaciaszek

Made another sample project that reproduces a problem. It also contains a submodule with proposed workaround by @DenisKorolev. There are also a test with boot 2.3 and ribbon that works fine and a test for when eager-loading enabled for Spring's loadbalancer (available in boot 3+, but you can implement the same logic for yourself if it fits).

use maven package to jar file then run jar , it get ClassNotFoundException

will not get ClassNotFoundException while not use jar file run

Dec 01 '22 06:12 SuperChrisliu

linux in docker had the problem in jdk17, but windows was ok in jdk17.

Feb 07 '23 06:02 poolsnowhui

I wanted to share some information about some similar bug.

We have one spring service working with docker, k8, java 17 and the use of CompletableFuture, everything was working ok, until we decided to add more memory and cpu

We pass from

resources:
  requests:
    memory: "64Mi"
    cpu: "48m"
  limits:
    memory: "512Mi"
    cpu: "1000m"

To

  resources:
    requests:
      memory: "512Mi"
      cpu: "60m"
    limits:
      memory: "512Mi"

And the same bug appear Could not find class [org.springframework.boot.autoconfigure.condition.OnBeanCondition] the classLoader was almost empty, like we pass from 15000 class to 50 in the classLoader.

But the the code was related to StreamBridge and not feign client (even if we use it in the code, but not directly in the code called), but for me is the same JVM/spring bug. Is was working ok with the IDE, but the Jar was failing, etc

The solution put in place was add

@Configuration
public class TaskExecutorConfig {

  @Bean("asyncPool")
  public Executor asyncPool() {

    return Executors.newFixedThreadPool(10);
  }
}

And pass it when we call the CompletableFuture

CompletableFuture.runAsync(
  () -> {
                your code here...
            },
  asyncPool
);

Feb 28 '23 10:02 nicolasArmando

Another workaround is manually init load balancer clients, for example after application ready event occured

@Configuration
@RequiredArgsConstructor
public class LoadBalancerClientsInitializer {

    private final LoadBalancerClientFactory loadBalancerClientFactory;

    @EventListener(ApplicationReadyEvent.class)
    public void onApplicationReadyEvent() {
        loadBalancerClientFactory.getInstance("client-name", NONE);
    }

}

@FeignClient(name = "client-name", contextId = "someClient")
public interface SomeClient {
}

Mar 31 '23 13:03 eldzej88

The previous workaround didn't work for me as the constructor of LoadBalancerClientFactory without arguments is deprecated, as a result that solution causes NPE.

@DenisKorolev is there any particular reason why you have omitted the following line from the previous workaround?
clientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));
This line still appears in the org.springframework.cloud.loadbalancer.config.LoadBalancerAutoConfiguration.

If you have a custom loadbalancer configure, like @LoadBalancerClients(defaultConfiguration = CustomLoadBalancerConfiguration.class) you should add it loadBalancerClientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));

    @Bean
    @ConditionalOnMissingBean
    public LoadBalancerClientFactory loadBalancerClientFactory(LoadBalancerClientsProperties properties, ObjectProvider<List<LoadBalancerClientSpecification>> configurations) {
        LoadBalancerClientFactory loadBalancerClientFactory = new LoadBalancerClientFactory(properties) {
            @Override
            protected AnnotationConfigApplicationContext createContext(String name) {
                // FIXME: temporary switch classloader to use the correct one when creating the context
                ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
                Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());
                AnnotationConfigApplicationContext context = super.createContext(name);
                Thread.currentThread().setContextClassLoader(originalClassLoader);
                return context;
            }
        };
        loadBalancerClientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));
        return loadBalancerClientFactory;
    }

Apr 15 '23 10:04 kisick

We experience the same issue with k8s resource limits. After we removed the CPU limit of 1, our application failed to execute Feign requests as the POD where the application runs now had more CPUs visible to the JVM. 4 instead of 1. I'm wondering why the Spring team does not see this as a serious issue as the usage of CompletableFuture in a project is very likely and we have numerous comments here with lots of related issues that were closed ...

Apr 17 '23 05:04 HJK181

Spring boot 3.1.0 spring-cloud 2022.0.3 Java 17

The previous workaround is working with little changes

 @Bean
  @ConditionalOnMissingBean
  public LoadBalancerClientFactory loadBalancerClientFactory(LoadBalancerClientsProperties properties) {
    return new LoadBalancerClientFactory(properties) {
      @Override public AnnotationConfigApplicationContext createContext(String name) {
        ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());
        AnnotationConfigApplicationContext context = (AnnotationConfigApplicationContext) super.createContext(name);
        Thread.currentThread().setContextClassLoader(originalClassLoader);
        return context;
      }
    };
  }

Jun 26 '23 11:06 badrkacimi

We got similar issue in one of our applications. After investigation, we found this is caused by https://bugs.openjdk.org/browse/JDK-8172726.

Start with Java 9, ForkJoinPool creates thread using system class loader as context class loader, refer to https://github.com/openjdk/jdk11u/blob/master/src/java.base/share/classes/java/util/concurrent/ForkJoinPool.java#L721.

If we run applications in IDE, the system class loader has been used to load Spring and application classes, so it's not easy to reproduce this issue. If we run applications from command line using fat JAR, org.springframework.boot.loader.LaunchedURLClassLoader will be used to load those classes instead. If threads switch to use system class loader, it will not be able to load Spring and other classes inside the fat JAR.

Jul 07 '23 06:07 crmky

The fix in https://github.com/spring-cloud/spring-cloud-commons/pull/1098 tries to use the class loader of parent ApplicationContext in child ApplicationContext:

context = new AnnotationConfigApplicationContext(beanFactory); context.setClassLoader(this.parent.getClassLoader());

However typically the parent ApplicationContext doesn't set class loader explicitly, it will use ClassUtils.getDefaultClassLoader() to return the default class loader. Unfortunately that implementation will return context class loader if that has been set.

Jul 07 '23 08:07 crmky

Our applications use a different workaround. We created ApplicationContextInitializer implementation (and added to spring.factories) to manually set the class loader as the context class loader who starts the application.

public class ClassLoaderApplicationContextInitializer
        implements ApplicationContextInitializer<ConfigurableApplicationContext> {

    @Override
    public void initialize(ConfigurableApplicationContext applicationContext) {
        applicationContext.setClassLoader(applicationContext.getClassLoader());
    }
}

Jul 07 '23 23:07 crmky

Thanks for looking into it @crmky. Have submitted a PR to address this in Commons.

Jul 19 '23 17:07 OlgaMaciaszek

Fixed with https://github.com/spring-cloud/spring-cloud-commons/pull/1256.

Jul 20 '23 08:07 OlgaMaciaszek

spring-cloud-openfeign spring-cloud-openfeign copied to clipboard

Feign client lazy initialization fails when using CompletableFuture.supplyAsync() to call the Feign client initially

spring-cloud-openfeign
spring-cloud-openfeign copied to clipboard