No such extension org.apache.dubbo.rpc.cluster.LoadBalance by name random

[winflex]

• [x] I have searched the issues of this repository and believe that this is not a duplicate.
• [x] I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

• Dubbo version: 2.7.3
• Operating System version: Linux version 3.10.0-957.27.2.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) )
• Java version: openjdk 11.0.2

Steps to reproduce this issue

    @Reference
    private UserService userService;
    @GetMapping("/v1/user/relation/trace/test2")
    public CompletableFuture<String> dubbo2Test() {
        return CompletableFuture.supplyAsync(() -> {
            return userService.getUserInfo("sam");
        }).thenApply(userInfo -> "s");
    }

2. dubbo config: it's very simple:

dubbo.registry.address=xxx
dubbo.reference.check=false

3. visit /v1/user/relation/trace/test2

Expected Result

"s"

Actual Result

2020-03-05 11:28:30.196 [ForkJoinPool.commonPool-worker-3] ERROR esa.restlight.core.bootstrap.DefaultDispatcherHandler - Error occurred when doing request(url=/v1/user/relation/trace/test2, method =GET)
java.util.concurrent.CompletionException: java.lang.IllegalStateException: No such extension org.apache.dubbo.rpc.cluster.LoadBalance by name random
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.lang.IllegalStateException: No such extension org.apache.dubbo.rpc.cluster.LoadBalance by name random
        at org.apache.dubbo.common.extension.ExtensionLoader.findException(ExtensionLoader.java:520)
        at org.apache.dubbo.common.extension.ExtensionLoader.createExtension(ExtensionLoader.java:527)
        at org.apache.dubbo.common.extension.ExtensionLoader.getExtension(ExtensionLoader.java:351)
        at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.initLoadBalance(AbstractClusterInvoker.java:296)
        at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:246)
        at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:78)
        at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:55)
        at org.apache.dubbo.common.bytecode.proxy0.getUserInfo(proxy0.java)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor$ReferenceBeanInvocationHandler.invoke(ReferenceAnnotationBeanPostProcessor.java:260)
        at com.sun.proxy.$Proxy72.getUserInfo(Unknown Source)
        at com.heytap.ugc.user.relation.rest.controller.TestController.lambda$dubbo2Test$34(TestController.java:182)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
        ... 6 common frames omitted

point: only when ForkJoinPool.commonPool thread call userService.getUserInfo can reproduce this issue

[winflex]

turns out that the ForJoinPool.commonPool's thread doesn't have the same context classloader as the thread which invokes CompletableFuture.xxxAsync()

still don't know the solution

[winflex]

Tried to customize my own ForkJoinPoolThreadFactory, and set by -Djava.util.concurrent.ForkJoinPool.common.threadFactory=MyFactory But it's not working in a spring boot application bacause ForkJoinPool uses AppClassLoader(returned by ClassLoader.getSystemClassLoader()) to load the MyFactory.class and ClassNotFoundException will be thrown. The rigth classloader would be spring boot's own classloader called LaunchedURLClassLoader

Still wondering!

[fibbery]

我觉得你可以用supplyAsync(Supplier<U> supplier,Executor executor)这个解决，自定义一个Executor来处理。 要么就是监听SpringApplicationEvent的各种事件，在监听处代码把ForkJoinPool.common.threadFactory这个final属性修改成自己的MyFactory

[winflex]

我觉得你可以用supplyAsync(Supplier supplier,Executor executor)这个解决，自定义一个Executor来处理。 要么就是监听SpringApplicationEvent的各种事件，在监听处代码把ForkJoinPool.common.threadFactory这个final属性修改成自己的MyFactory

两种都考虑过，我就想问问有没有更优雅的方式

[niuchen]

遇到同样的问题 不知道可以解决吗

[niuchen]

Caused by: java.lang.IllegalStateException: No such extension org.apache.dubbo.rpc.cluster.LoadBalance by name random at org.apache.dubbo.common.extension.ExtensionLoader.findException(ExtensionLoader.java:623) at org.apache.dubbo.common.extension.ExtensionLoader.createExtension(ExtensionLoader.java:630) at org.apache.dubbo.common.extension.ExtensionLoader.getExtension(ExtensionLoader.java:429) at org.apache.dubbo.common.extension.ExtensionLoader.getExtension(ExtensionLoader.java:413) at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.initLoadBalance(AbstractClusterInvoker.java:308) at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:258) at org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47) at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92) at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:88) at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74) at org.apache.dubbo.common.bytecode.proxy0.currentTaskInfos(proxy0.java) at com.elitesland.pur.service.impl.PurPrServiceImpl.taskName(PurPrServiceImpl.java:156)

[niuchen]

服务器服务开始是好的. 不一定什么时候就一直报错. 只能重启服务才可以恢复. 是有大内存消耗,CG释放了缓存吗? 还是其他的原因. 对dubbo感觉不可靠了.

[SimpleLym]

在使用CompletableFuture时使用自定义线程池可以解决这个问题；或者在调用dubbo服务前手动设置当前线程的classLoader；两种方案均可；

[winlans]

为什么这个issue还没有被关闭，难道最新版还没有修复？

[kiss-keray]

为什么这个issue还没有被关闭，难道最新版还没有修复？

找到解决方法没啊？同运行一段时间就找不到LoadBalance了

[AlbumenJ]

为什么这个issue还没有被关闭，难道最新版还没有修复？

找到解决方法没啊？同运行一段时间就找不到LoadBalance了

考虑是否有热部署等场景动态切换 Classloader 的

[kiss-keray]

为什么这个issue还没有被关闭，难道最新版还没有修复？

找到解决方法没啊？同运行一段时间就找不到LoadBalance了

考虑是否有热部署等场景动态切换 Classloader 的

不考虑，直接自定义了Cluster 重写initLoadBalance方法解决问题了

[guoxianxian]

服务器服务开始是好的. 不一定什么时候就一直报错. 只能重启服务才可以恢复. 是有大内存消耗,CG释放了缓存吗? 还是其他的原因. 对dubbo感觉不可靠了.

请问找到解决方案了吗，我试过自定义loadbalance也不管用

[zhuxiaoge888]

自定义线程就好了 ，不要用默认的forkjoin

[akhe-h]

这个issue还没解决吗？遇到同样的问题，但重启一下又好了。。。。。 Dubbo Spring Boot (v2.7.3) Dubbo (v2.7.3) :

[korisky]

For those still using Dubbo 2.7.x and encountering similar issues, there is a workaround involving Pre-loading of necessary SPIs by a thread with the 'correct' class loader:

If you're using SpringBoot integrated with Dubbo, you can add the following code right after the service is ready(e.g. attach it with the SmartLiftcycle interface). This will ensure that the code is executed by a Spring-managed thread, which uses the same class loader as Dubbo:

Pre-load solution

You may also need to set serialization manually: dubbo.protocol.serialization=hessian2

@Slf4j
@Configuration
public class CustomLifeCycle implements SmartLifecycle {

    @Override
    public void start() {
        // under executable .jar file, the classloader for this thread would be 'LaunchedURLClassLoader', 
        // the top classloader for Spring Application
        ExtensionLoader.getExtensionLoader(LoadBalance.class)
                .getDefaultExtensionName();
    }

    @Override
    public void stop() {
    }

    @Override
    public boolean isRunning() {
        return false;
    }
}

What cause this?

Under Dubbo 2.7.17 version, the source code of org.apache.dubbo.common.extension.ExtensionLoader#getExtensionClasses is taking the Dubbo-Check-Lock(DCL) strategy, and Dubbo only loads needed SPI when needed (similar to lazy initialization):

private Map<String, Class<?>> getExtensionClasses() {
    Map<String, Class<?>> classes = cachedClasses.get();
    if (classes == null) {
        synchronized (cachedClasses) {
            classes = cachedClasses.get();
            if (classes == null) {
                classes = loadExtensionClasses(); // loadExtensioClasses() would never return NULL
                cachedClasses.set(classes);
            }
        }
    }
    return classes;
}

As the replies above mentioned, inner logic for org.apache.dubbo.common.extension.ExtensionLoader#loadExtensionClasses

/**
 * synchronized in getExtensionClasses
 */
private Map<String, Class<?>> loadExtensionClasses() {
    cacheDefaultExtensionName();

    // never return NULL
    Map<String, Class<?>> extensionClasses = new HashMap<>();

    for (LoadingStrategy strategy : strategies) {
        // inside the loadDirectory function's logic, it would call classloader
        loadDirectory(extensionClasses, strategy.directory(), type.getName(), strategy.preferExtensionClassLoader(),
                strategy.overridden(), strategy.excludedPackages());
        loadDirectory(extensionClasses, strategy.directory(), type.getName().replace("org.apache", "com.alibaba"),
                strategy.preferExtensionClassLoader(), strategy.overridden(), strategy.excludedPackages());
    }

    return extensionClasses;
}

would use calling thread's class loader in org.apache.dubbo.common.utils.ClassUtils#getClassLoader(java.lang.Class<?>)

public static ClassLoader getClassLoader(Class<?> clazz) {
    ClassLoader cl = null;
    try {
        // For forkJoinPool's thread, the contextClassLoader would be: AppClassLoader (JDK9+)
        // But Dubbo with Spring integration is under Spring's handling, which means we could load Dubbo-implemented SPIs with classloader: LaunchedURLClassLoader (or classloaders's parent is it)
        cl = Thread.currentThread().getContextClassLoader();
    } catch (Throwable ex) {
        // Cannot access thread context ClassLoader - falling back to system class loader...
    }
    if (cl == null) {
        // No thread context class loader -> use class loader of this class.
        cl = clazz.getClassLoader();
        if (cl == null) {
            // getClassLoader() returning null indicates the bootstrap ClassLoader
            try {
                cl = ClassLoader.getSystemClassLoader();
            } catch (Throwable ex) {
                // Cannot access system ClassLoader - oh well, maybe the caller can live with null...
            }
        }
    }

    return cl;
}

So the first Thread call the getExtensionClasses is crucial, or else incorrect classloader return would affect all later requests (loadExtensionClasses would never return null).

Why the workaround works?

Thus, the above workaround is using the Spring-handled thread (classloader would be LaunchedURLClassLoader in the executable jar file) to initialise that cachedClasses because:

public String getDefaultExtensionName() {
    // Let the Spring-handled thread init the DCL code
    getExtensionClasses();
    return cachedDefaultName;
}

Did Dubbo 3.x solve this?

They do make some changes in the ClassUtils (3.2 ver), by the class which start with "org.apache.dubbo", they just use the target class's classloader rather than get if from current Thread.

public static ClassLoader getClassLoader(Class<?> clazz) {
        ClassLoader cl = null;
        if (!clazz.getName().startsWith("org.apache.dubbo")) {
            cl = clazz.getClassLoader();
        }
        if (cl == null) {
            try {
                cl = Thread.currentThread().getContextClassLoader();
            } catch (Exception ignored) {
                // Cannot access thread context ClassLoader - falling back to system class loader...
            }
            if (cl == null) {
                // No thread context class loader -> use class loader of this class.
                cl = clazz.getClassLoader();
                if (cl == null) {
                    // getClassLoader() returning null indicates the bootstrap ClassLoader
                    try {
                        cl = ClassLoader.getSystemClassLoader();
                    } catch (Exception ignored) {
                        // Cannot access system ClassLoader - oh well, maybe the caller can live with null...
                    }
                }
            }
        }

        return cl;
    }

Hope this comment helps anyone struggling with this :)