eventmesh
eventmesh copied to clipboard
apache
[Question] How to use kafka as a storage layer
Search before asking
- [X] I had searched in the issues and found no similar issues.
Question
When I want to use kafka as the storage layer for eventmesh,the runtime start fail.
The configuration is as follow: #connector plugin eventMesh.connector.plugin.type=standalone
#storage plugin eventMesh.storage.plugin.type=kafka
Does eventmesh support kafka now? If support,how to start eventmesh with kafka?
Welcome to the Apache EventMesh community!! We are glad that you are contributing by opening this issue. :D
Please make sure to include all the relevant context. We will be here shortly.
If you are interested in contributing to our project, please let us know! You can check out our contributing guide on contributing to EventMesh.
Want to get closer to the community?
| WeChat Assistant | WeChat Public Account | Slack |
|---|---|---|
 |
 |
Join Slack Chat |
Mailing Lists:
| Name | Description | Subscribe | Unsubscribe | Archive |
|---|---|---|---|---|
| Users | User support and questions mailing list | Subscribe | Unsubscribe | Mail Archives |
| Development | Development related discussions | Subscribe | Unsubscribe | Mail Archives |
| Commits | All commits to repositories | Subscribe | Unsubscribe | Mail Archives |
| Issues | Issues or PRs comments and reviews | Subscribe | Unsubscribe | Mail Archives |
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Kafka storage plugin is supported. There is a Kafka connector PR still WIP. You may look for its progress here: https://github.com/apache/eventmesh/pull/4180.
First, it will be better if you can modify this issue title to something concrete. "Question title" is a bad title.
Then, Kafka storage plugin is supported. You can provide the error stack information for community accurately locating your problem.
eventMesh.storage.plugin.type=kafka
still facing this issue when run
java -Deventmesh.log.home=/Users/tuan/Desktop/apache-eventmesh-1.9.0 -DconfPath=${EVENTMESH_HOME}/conf -classpath ${EVENTMESH_HOME}/conf:${EVENTMESH_HOME}/apps/:${EVENTMESH_HOME}/lib/ org.apache.eventmesh.runtime.boot.EventMeshStartup
3-09-15 00:31:00,407 DEBUG [grpc-nio-boss-ELG-1-1] DefaultChannelId(DefaultChannelId.java:79) - -Dio.netty.processId: 51938 (auto-detected)
2023-09-15 00:31:00,409 DEBUG [grpc-nio-boss-ELG-1-1] NetUtil(NetUtil.java:135) - -Djava.net.preferIPv4Stack: false
2023-09-15 00:31:00,409 DEBUG [grpc-nio-boss-ELG-1-1] NetUtil(NetUtil.java:136) - -Djava.net.preferIPv6Addresses: false
2023-09-15 00:31:00,411 DEBUG [grpc-nio-boss-ELG-1-1] NetUtilInitializations(NetUtilInitializations.java:129) - Loopback interface: lo0 (lo0, 0:0:0:0:0:0:0:1%lo0)
2023-09-15 00:31:00,412 DEBUG [grpc-nio-boss-ELG-1-1] NetUtil(NetUtil.java:187) - Failed to get SOMAXCONN from sysctl and file /proc/sys/net/core/somaxconn. Default: 128
2023-09-15 00:31:00,414 DEBUG [grpc-nio-boss-ELG-1-1] DefaultChannelId(DefaultChannelId.java:101) - -Dio.netty.machineId: 4c:32:75:ff:fe:92:a4:71 (auto-detected)
2023-09-15 00:31:00,439 DEBUG [grpc-nio-boss-ELG-1-1] ByteBufUtil(ByteBufUtil.java:87) - -Dio.netty.allocator.type: pooled
2023-09-15 00:31:00,439 DEBUG [grpc-nio-boss-ELG-1-1] ByteBufUtil(ByteBufUtil.java:96) - -Dio.netty.threadLocalDirectBufferSize: 0
2023-09-15 00:31:00,439 DEBUG [grpc-nio-boss-ELG-1-1] ByteBufUtil(ByteBufUtil.java:99) - -Dio.netty.maxThreadLocalCharBufferSize: 16384
2023-09-15 00:31:00,452 INFO [main] EventMeshGrpcServer(EventMeshGrpcServer.java:150) - ---------------EventMeshGRPCServer running-------------------
2023-09-15 00:31:00,478 INFO [main] JarExtensionClassLoader(JarExtensionClassLoader.java:132) - load extension class success, extensionType: interface org.apache.eventmesh.api.admin.Admin, extensionClass: class org.apache.eventmesh.storage.rocketmq.admin.RocketMQAdminAdaptor
2023-09-15 00:31:00,479 INFO [main] JarExtensionClassLoader(JarExtensionClassLoader.java:132) - load extension class success, extensionType: interface org.apache.eventmesh.api.admin.Admin, extensionClass: class org.apache.eventmesh.storage.standalone.admin.StandaloneAdminAdaptor
2023-09-15 00:31:00,480 INFO [main] JarExtensionClassLoader(JarExtensionClassLoader.java:132) - load extension class success, extensionType: interface org.apache.eventmesh.api.admin.Admin, extensionClass: class org.apache.eventmesh.storage.rocketmq.admin.RocketMQAdminAdaptor
2023-09-15 00:31:00,480 INFO [main] JarExtensionClassLoader(JarExtensionClassLoader.java:132) - load extension class success, extensionType: interface org.apache.eventmesh.api.admin.Admin, extensionClass: class org.apache.eventmesh.storage.standalone.admin.StandaloneAdminAdaptor
2023-09-15 00:31:00,480 ERROR [main] MQAdminWrapper(MQAdminWrapper.java:40) - can't load the meshMQAdmin plugin, please check.
2023-09-15 00:31:00,485 pool-1-thread-1 DEBUG Stopping LoggerContext[name=1b6d3586, org.apache.logging.log4j.core.LoggerContext@428640fa]
2023-09-15 00:31:00,486 pool-1-thread-1 DEBUG Stopping LoggerContext[name=1b6d3586, org.apache.logging.log4j.core.LoggerContext@428640fa]...
2023-09-15 00:31:00,480 ERROR [main] EventMeshStartup(EventMeshStartup.java:57) - EventMesh start fail.
java.lang.RuntimeException: doesn't load the meshMQAdmin plugin, please check.
at org.apache.eventmesh.runtime.core.plugin.MQAdminWrapper.
@nguyenminhtuanfit Please use the lastest master branch version of EventMesh. It has been fixed.
Have you encountered any additional problems related to this issue? Please share any relevant experiences here. If you think your issue has been resolved, please feel free to close this thread.
It has been 90 days since the last activity on this issue. Apache EventMesh values the voices of the community. Please don't hesitate to share your latest insights on this matter at any time, as the community is more than willing to engage in discussions regarding the development and optimization directions of this feature.
If you feel that your issue has been resolved, please feel free to close it. Should you have any additional information to share, you are welcome to reopen this issue.