rocketmq-helm icon indicating copy to clipboard operation
rocketmq-helm copied to clipboard

老哥救救我 help me,help me

Open 743157 opened this issue 1 year ago • 16 comments

在启动proxy的pod时报错: 03:12:56,493 |-INFO in org.apache.rocketmq.common.logging.JoranConfiguratorExt@33723e30 - Registering current configuration as safe fallback point

org.apache.rocketmq.proxy.common.ProxyException: create system broadcast topic DefaultHeartBeatSyncerTopic failed on cluster rocketmq at org.apache.rocketmq.proxy.service.sysmessage.AbstractSystemMessageSyncer.createSysTopic(AbstractSystemMessageSyncer.java:177) at org.apache.rocketmq.proxy.service.sysmessage.AbstractSystemMessageSyncer.start(AbstractSystemMessageSyncer.java:143) at org.apache.rocketmq.proxy.service.client.ClusterConsumerManager.start(ClusterConsumerManager.java:68) at org.apache.rocketmq.common.utils.AbstractStartAndShutdown.start(AbstractStartAndShutdown.java:33) at org.apache.rocketmq.common.utils.AbstractStartAndShutdown.start(AbstractStartAndShutdown.java:33) at org.apache.rocketmq.common.utils.AbstractStartAndShutdown.start(AbstractStartAndShutdown.java:33) at org.apache.rocketmq.proxy.ProxyStartup.main(ProxyStartup.java:96),实在没找到原因用的GKE集群

743157 avatar May 20 '24 03:05 743157

ns和broker都起来了且没有报错信息

743157 avatar May 20 '24 03:05 743157

proxy 依赖 ns 和 broker,等其他都启好了再看看 proxy

itboon avatar May 20 '24 03:05 itboon

proxy 依赖 ns 和 broker,等其他都启好了再看看 proxy

老哥都起来了,没有报错信息,就只有proxy起不来

743157 avatar May 20 '24 03:05 743157

可能是我描述问题不太清楚,我在搭建的时候出现了 create system broadcast topic DefaultHeartBeatSyncerTopic failed on cluster rocketmqnt 这个报错来自于rocketmq-proxy而现在另外的组件broker和nameserver 的状态都是running,只有proxy报了这一个错误,我实在没有找到原因.我目前只排查了通讯问题

743157 avatar May 20 '24 06:05 743157

请问你是怎么部署的,麻烦提供完整的 helm install 命令,如果有 values 也请提供一下。另外 proxy 是可选组件,如果用不到可以去掉。

itboon avatar May 20 '24 06:05 itboon

好的 ,安装命令是 helm install rocketmq-test -nrocketmq .

743157 avatar May 20 '24 06:05 743157

values文件 :clusterName: "DefaultCluster"

image: repository: "apache/rocketmq" pullPolicy: Always tag: "5.2.0"

broker: size: master: 1 replica: 1

master: brokerRole: ASYNC_MASTER jvm: maxHeapSize: 2048M # javaOptsOverride: "" resources: limits: cpu: 4 memory: 16Gi requests: cpu: 200m memory: 2Gi

replica: jvm: maxHeapSize: 1024M # javaOptsOverride: "" resources: limits: cpu: 4 memory: 16Gi requests: cpu: 50m memory: 2Gi

persistence: enabled: false size: 20Gi #storageClass: "gp2"

aclConfigMapEnabled: false aclConfig: | globalWhiteRemoteAddresses: - '' - 10... - 192.168..

config: ## brokerClusterName brokerName brokerRole brokerId 由内置脚本自动生成 deleteWhen: "04" fileReservedTime: "48" flushDiskType: "ASYNC_FLUSH" waitTimeMillsInSendQueue: "1000" # aclEnable: true

affinityOverride: {} tolerations: [] nodeSelector: role: "node2"

broker.readinessProbe

readinessProbe: tcpSocket: port: main initialDelaySeconds: 20 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3

nameserver: replicaCount: 1

jvm: maxHeapSize: 600M # javaOptsOverride: ""

resources: limits: cpu: 2 memory: 6Gi ephemeral-storage: 8Gi requests: cpu: 100m memory: 1Gi ephemeral-storage: 1Gi

persistence: enabled: false size: 8Gi #storageClass: "gp2"

affinityOverride: {} tolerations: [] nodeSelector: role: "node2"

nameserver.readinessProbe

readinessProbe: tcpSocket: port: main initialDelaySeconds: 20 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3

proxy: enabled: true replicaCount: 1 jvm: maxHeapSize: 800M # javaOptsOverride: ""

resources: limits: cpu: 2 memory: 6Gi requests: cpu: 600m memory: 2Gi

affinityOverride: {} tolerations: [] nodeSelector: role: "node2"

proxy.readinessProbe

readinessProbe: tcpSocket: port: main initialDelaySeconds: 20 periodSeconds: 20 timeoutSeconds: 10 failureThreshold: 2

proxy.service

service: annotations: {} type: ClusterIP

dashboard: enabled: false replicaCount: 1 image: repository: "apacherocketmq/rocketmq-dashboard" pullPolicy: IfNotPresent tag: "1.0.0"

jvm: maxHeapSize: 256M

resources: limits: cpu: 1 memory: 2Gi requests: cpu: 20m memory: 512Mi

dashboard.readinessProbe

readinessProbe: failureThreshold: 5 httpGet: path: / port: http livenessProbe: {}

service: annotations: {} type: ClusterIP nodePort: 31007

ingress: enabled: false className: "" annotations: {} # nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/8,124.160.30.50 hosts: - host: rocketmq-dashboard.example.com tls: [] # - secretName: example-tls # hosts: # - rocketmq-dashboard.example.com 就是为了要用proxy才安装的

743157 avatar May 20 '24 06:05 743157

我改动得很少,甚至都没做持久化,我想先搭建起来才做持久化等等细节

743157 avatar May 20 '24 06:05 743157

您请忽略clusterName: "DefaultCluster" 这一项 原本是clusterName: "rocketmq"

743157 avatar May 20 '24 06:05 743157

你可以按文档先部署一个最小化的集群试试:

helm repo add rocketmq-repo https://helm-charts.itboon.top/rocketmq
helm repo update rocketmq-repo

helm upgrade --install rocketmq \
  --namespace rocketmq-demo \
  --create-namespace \
  --set broker.persistence.enabled="false" \
  rocketmq-repo/rocketmq

itboon avatar May 20 '24 07:05 itboon

试过了 同样的报错

743157 avatar May 20 '24 07:05 743157

估计没了

743157 avatar May 28 '24 01:05 743157

我在nameserver通过手敲命令创建topic显示成功 但是我通过命令sendMessage 失败

743157 avatar May 28 '24 08:05 743157

老哥 找到原因了 希望你修复一下,也有可能只是在谷歌GKE集群上的问题 ,你的templates的rocketmq-broker和rocketmq-proxy和dashboard的获取nameser的地址的env不对,在执行helm install rocketmq -nrocketmq时没有获取到真实的nameserver地址而是rocketmq-nameserver-0.rocketmq-nameserver:9876 这个地址是错的 导致broker注册不了ns,而proxy也是一样的 注册不了,在deployment和satafulset里面的 env: - name: NAMESRV_ADDR value: {{ include "rocketmq.nameserver.addr" . }} #rocketmq-nameserver-0.rocketmq-nameserver:9876 建议改成nameserver的service地址

743157 avatar May 30 '24 11:05 743157

还有dashboard的一个java启动命令里面也是错的

743157 avatar May 30 '24 11:05 743157

我也遇到了同样的问题。 但我部署最小化集群的时候,proxy重启了几次 就成功了,可是部署集群老卡在这个问题上 求楼主解决

yxp1350484629 avatar Aug 06 '24 07:08 yxp1350484629

请使用 12.3.0 及以上版本部署,老版本在某些集群版本存在 bug,如果仍然有问题请重新提交 issue

itboon avatar Aug 25 '24 08:08 itboon

在启动proxy的pod时报错: org.apache.rocketmq.proxy.common.ProxyException: create system broadcast topic DefaultHeartBeatSyncerTopic failed on cluster DefaultCluster

我也遇到了同样的问题,怀疑是开启了ACL导致,不知道怎么解决

# 集群名
clusterName: "DefaultCluster"

image:
  repository: "docker.libcuda.so/apache/rocketmq"
  tag: "5.3.2"

broker:
  aclConfigMapEnabled: true
  config:
    aclEnable: true
  aclConfig: |
    accounts:
      - accessKey: admin123
        secretKey: admin123
        admin: true
  # 3个 master 节点,每个 master 具有1个副节点,共6个 broker 节点
  size:
    master: 3
    replica: 1

  persistence:
    enabled: true
    size: 20Gi
    storageClass: integrate

  # 主节点资源分配
  master:
    brokerRole: ASYNC_MASTER
    jvm:
      maxHeapSize: 4096M
    resources:
      limits:
        cpu: 4
        memory: 4Gi
      requests:
        cpu: 500m
        memory: 2Gi

  # 副节点资源分配
  replica:
    jvm:
      maxHeapSize: 2048M
    resources:
      limits:
        cpu: 4
        memory: 4Gi
      requests:
        cpu: 100m
        memory: 2Gi

nameserver:
  replicaCount: 3
  jvm:
    maxHeapSize: 1024M
  resources:
    limits:
      cpu: 4
      memory: 4Gi
    requests:
      cpu: 50m
      memory: 1Gi

dashboard:
  enabled: true
  replicaCount: 1
  image:
    repository: "docker.libcuda.so/apacherocketmq/rocketmq-dashboard"
    pullPolicy: IfNotPresent
    tag: "2.0.1"
  auth:
    enabled: true
    users:
      - name: admin
        password: admin123
        isAdmin: true

  ingress:
    enabled: true
    className: "nginx"
    hosts:
      - host: rocketmq-dashboard.example.com
    tls: [ ]
    #  - secretName: example-tls
    #    hosts:
    #      - rocketmq-dashboard.example.com

proxy:
  service:
    annotations: { }
    type: NodePort
    nodePort: 8080
    nodePortGrpc: 8081

lnnt avatar Apr 26 '25 15:04 lnnt

create system broadcast topic DefaultHeartBeatSyncerTopic failed on cluster

参考 https://github.com/apache/rocketmq/issues/6486 把tools.yml挂载到proxy容器内就好了

lnnt avatar Apr 26 '25 16:04 lnnt