tugraph-analytics icon indicating copy to clipboard operation
tugraph-analytics copied to clipboard
verified publisher icon TuGraph-family

GQLAlgorithmTest fails in Windows

Open liuxiaocs7 opened this issue 2 years ago • 2 comments

Describe the bug A clear and concise description of what the bug is.

GQLAlgorithmTest fails in Windows but works in Ubuntu, logs:

2023-06-27 02:08:47 INFO  EnvironmentFactory:104 - loaded IEnvironmentArgsParser implementation com.antgroup.geaflow.env.args.EnvironmentArgumentParser@6337c201
2023-06-27 02:08:47 INFO  EnvironmentFactory:84 - loaded IEnvironment implementation com.antgroup.geaflow.cluster.local.client.LocalEnvironment@6304101a
2023-06-27 02:08:47 INFO  GQLPipeLine:85 - queryPath:resource:///query/gql_algorithm_001.sql
2023-06-27 02:08:47 INFO  GQLPipeLine:99 - execute query:
CREATE TABLE console (
  vid int,
	distance bigint
) WITH (
	type='file',
	geaflow.dsl.file.path='file://D:\Development\Code\Repository\tugraph-analytics\geaflow\geaflow-dsl\geaflow-dsl-runtime\target/gql_algorithm_001'
);

USE GRAPH modern;

INSERT INTO console
CALL SSSP(1) YIELD (vid, distance)
RETURN cast (vid as int), distance
;
2023-06-27 02:08:47 INFO  GQLPipeLine:104 - Submit pipeline task ...
2023-06-27 02:08:48 INFO  GQLPipeLine:109 - Execute pipeline task
2023-06-27 02:08:48 WARN  MetricGroupRegistry:90 - report list is empty
2023-06-27 02:08:48 WARN  HAServiceFactory:55 - unknown ha service type:memory, use default memoryHaService
2023-06-27 02:08:48 INFO  ClusterMetaStoreFactory:55 - create memory cluster metastore
2023-06-27 02:08:48 ERROR LocalClusterClient:55 - deploy cluster failed
com.antgroup.geaflow.common.exception.GeaflowRuntimeException: 
************
ERR_ID:
     RUN-00000003
CAUSE:
     SystemInternalError - not find 'MEMORY' SPI implement
ACTION:
     Please contact admin.
DETAIL:

************
	at com.antgroup.geaflow.store.api.key.StoreBuilderFactory.build(StoreBuilderFactory.java:40) ~[classes/:?]
	at com.antgroup.geaflow.cluster.system.MemoryClusterMetaKVStore.init(MemoryClusterMetaKVStore.java:30) ~[classes/:?]
	at com.antgroup.geaflow.cluster.system.ClusterMetaStoreFactory.create(ClusterMetaStoreFactory.java:43) ~[classes/:?]
	at com.antgroup.geaflow.cluster.system.ClusterMetaStore.<init>(ClusterMetaStore.java:49) ~[classes/:?]
	at com.antgroup.geaflow.cluster.system.ClusterMetaStore.init(ClusterMetaStore.java:55) ~[classes/:?]

Expected behavior A clear and concise description of what you expected to happen.

tests pass

Additional context Add any other context about the problem here.

In this method, it seems that only load jdbc-store from SPI, but not includes rocksdb, memory and redis in window.

public static synchronized IStoreBuilder build(String storeType) {
    if (CONCURRENT_TYPE_MAP.containsKey(storeType)) {
        return CONCURRENT_TYPE_MAP.get(storeType);
    }

    ServiceLoader<IStoreBuilder> serviceLoader = ServiceLoader.load(IStoreBuilder.class);
    for (IStoreBuilder storeBuilder: serviceLoader) {
        if (storeBuilder.getStoreDesc().name().equalsIgnoreCase(storeType)) {
            CONCURRENT_TYPE_MAP.put(storeType, storeBuilder);
            return storeBuilder;
        }
    }
    throw new GeaflowRuntimeException(RuntimeErrors.INST.spiNotFoundError(storeType));
}

liuxiaocs7 avatar Jun 26 '23 18:06 liuxiaocs7

Using the latest code(https://github.com/TuGraph-family/tugraph-analytics/commit/da00170ff6cea0a226a28a1eb6fd99bcd720f301) from the master branch, the problem seems to still exist. CC: @fanzhidongyzby

2024-01-23 02:11:43 [shuffle-writer-11-Message] INFO  PipelineOutputEmitter:198 - emitter task finish window id 1 11[0/2]
2024-01-23 02:11:43 [shuffle-writer-10-Message] INFO  PipelineOutputEmitter:198 - emitter task finish window id 1 12[1/2]
2024-01-23 02:11:43 [geaflow-worker-11] ERROR AbstractTaskRunner:48 - Error in create file: D:DevelopmentCodeRepository	ugraph-analyticsgeaflowgeaflow-dslgeaflow-dsl-runtime	arget\gql_algorithm_001\partition_0
com.antgroup.geaflow.dsl.common.exception.GeaFlowDSLException: Error in create file: D:DevelopmentCodeRepository	ugraph-analyticsgeaflowgeaflow-dslgeaflow-dsl-runtime	arget\gql_algorithm_001\partition_0
	at com.antgroup.geaflow.dsl.connector.file.sink.LocalFileWriteHandler.init(LocalFileWriteHandler.java:62)
	at com.antgroup.geaflow.dsl.connector.file.sink.FileTableSink.open(FileTableSink.java:53)
	at com.antgroup.geaflow.dsl.connector.api.function.GeaFlowTableSinkFunction.open(GeaFlowTableSinkFunction.java:55)
	at com.antgroup.geaflow.operator.base.AbstractOperator.open(AbstractOperator.java:101)
	at com.antgroup.geaflow.operator.base.window.AbstractStreamOperator.open(AbstractStreamOperator.java:37)
	at com.antgroup.geaflow.operator.impl.window.SinkOperator.open(SinkOperator.java:39)
	at com.antgroup.geaflow.operator.base.AbstractOperator.open(AbstractOperator.java:106)
	at com.antgroup.geaflow.operator.base.window.AbstractStreamOperator.open(AbstractStreamOperator.java:37)
	at com.antgroup.geaflow.operator.base.AbstractOperator.open(AbstractOperator.java:106)
	at com.antgroup.geaflow.operator.base.window.AbstractStreamOperator.open(AbstractStreamOperator.java:37)
	at com.antgroup.geaflow.operator.impl.window.FlatMapOperator.open(FlatMapOperator.java:32)
	at com.antgroup.geaflow.processor.impl.AbstractProcessor.open(AbstractProcessor.java:53)
	at com.antgroup.geaflow.processor.impl.AbstractStreamProcessor.open(AbstractStreamProcessor.java:49)
	at com.antgroup.geaflow.runtime.core.worker.AbstractComputeWorker.open(AbstractComputeWorker.java:44)
	at com.antgroup.geaflow.runtime.core.protocol.InitCycleEvent.execute(InitCycleEvent.java:68)
	at com.antgroup.geaflow.cluster.task.Task.execute(Task.java:36)
	at com.antgroup.geaflow.cluster.task.runner.TaskRunner.process(TaskRunner.java:54)
	at com.antgroup.geaflow.cluster.task.runner.TaskRunner.process(TaskRunner.java:24)
	at com.antgroup.geaflow.cluster.task.runner.AbstractTaskRunner.run(AbstractTaskRunner.java:43)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: D:DevelopmentCodeRepository	ugraph-analyticsgeaflowgeaflow-dslgeaflow-dsl-runtime	arget\gql_algorithm_001\partition_0 (文件名、目录名或卷标语法不正确。)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
	at java.io.FileWriter.<init>(FileWriter.java:90)
	at com.antgroup.geaflow.dsl.connector.file.sink.LocalFileWriteHandler.init(LocalFileWriteHandler.java:58)
	... 21 more
2024-01-23 02:11:43 [geaflow-worker-11] ERROR ComponentUncaughtExceptionHandler:27 - FATAL exception in thread:geaflow-worker-11
com.antgroup.geaflow.common.exception.GeaflowRuntimeException: com.antgroup.geaflow.dsl.common.exception.GeaFlowDSLException: Error in create file: D:DevelopmentCodeRepository	ugraph-analyticsgeaflowgeaflow-dslgeaflow-dsl-runtime	arget\gql_algorithm_001\partition_0
	at com.antgroup.geaflow.cluster.task.runner.AbstractTaskRunner.run(AbstractTaskRunner.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.antgroup.geaflow.dsl.common.exception.GeaFlowDSLException: Error in create file: D:DevelopmentCodeRepository	ugraph-analyticsgeaflowgeaflow-dslgeaflow-dsl-runtime	arget\gql_algorithm_001\partition_0
	at com.antgroup.geaflow.dsl.connector.file.sink.LocalFileWriteHandler.init(LocalFileWriteHandler.java:62)
	at com.antgroup.geaflow.dsl.connector.file.sink.FileTableSink.open(FileTableSink.java:53)
	at com.antgroup.geaflow.dsl.connector.api.function.GeaFlowTableSinkFunction.open(GeaFlowTableSinkFunction.java:55)
	at com.antgroup.geaflow.operator.base.AbstractOperator.open(AbstractOperator.java:101)
	at com.antgroup.geaflow.operator.base.window.AbstractStreamOperator.open(AbstractStreamOperator.java:37)
	at com.antgroup.geaflow.operator.impl.window.SinkOperator.open(SinkOperator.java:39)
	at com.antgroup.geaflow.operator.base.AbstractOperator.open(AbstractOperator.java:106)
	at com.antgroup.geaflow.operator.base.window.AbstractStreamOperator.open(AbstractStreamOperator.java:37)
	at com.antgroup.geaflow.operator.base.AbstractOperator.open(AbstractOperator.java:106)
	at com.antgroup.geaflow.operator.base.window.AbstractStreamOperator.open(AbstractStreamOperator.java:37)
	at com.antgroup.geaflow.operator.impl.window.FlatMapOperator.open(FlatMapOperator.java:32)
	at com.antgroup.geaflow.processor.impl.AbstractProcessor.open(AbstractProcessor.java:53)
	at com.antgroup.geaflow.processor.impl.AbstractStreamProcessor.open(AbstractStreamProcessor.java:49)
	at com.antgroup.geaflow.runtime.core.worker.AbstractComputeWorker.open(AbstractComputeWorker.java:44)
	at com.antgroup.geaflow.runtime.core.protocol.InitCycleEvent.execute(InitCycleEvent.java:68)
	at com.antgroup.geaflow.cluster.task.Task.execute(Task.java:36)
	at com.antgroup.geaflow.cluster.task.runner.TaskRunner.process(TaskRunner.java:54)
	at com.antgroup.geaflow.cluster.task.runner.TaskRunner.process(TaskRunner.java:24)
	at com.antgroup.geaflow.cluster.task.runner.AbstractTaskRunner.run(AbstractTaskRunner.java:43)
	... 3 more
Caused by: java.io.FileNotFoundException: D:DevelopmentCodeRepository	ugraph-analyticsgeaflowgeaflow-dslgeaflow-dsl-runtime	arget\gql_algorithm_001\partition_0 (文件名、目录名或卷标语法不正确。)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
	at java.io.FileWriter.<init>(FileWriter.java:90)
	at com.antgroup.geaflow.dsl.connector.file.sink.LocalFileWriteHandler.init(LocalFileWriteHandler.java:58)
	... 21 more
2024-01-23 02:11:43 [geaflow-worker-11] INFO  MemoryStatsWriter:25 - update metric: key=geaflowc62334e4-3863-44d3-b56d-cc69dca36481_exception_9223370330907671944, value=com.antgroup.geaflow.stats.model.ExceptionInfo@73563200

liuxiaocs7 avatar Jan 22 '24 18:01 liuxiaocs7

We've noticed that the implementation of MEMORY SPI is located within the geaflow-plugins package. Please ensure that you have correctly installed this package into your local Maven repository. Navigate to the directory containing the source code of geaflow-plugins. Execute the following Maven command to install the geaflow-plugins package:

mvn clean install

This command will compile and install geaflow-plugins into your local Maven repository. Once the installation is complete, please try rerunning your Java program to see if it resolves the issue you encountered while running on Windows.

Leomrlin avatar Jan 24 '24 03:01 Leomrlin