netmaker icon indicating copy to clipboard operation
netmaker copied to clipboard

Published 20 hours ago • verified publisher icon gravitl

[Bug]: After a while, it always needs to be manually click "Refresh Hosts Keys"

Open liuweiGL opened this issue 9 months ago • 11 comments

Contact Details

[email protected]

What happened?

aliyun-gw has a static public ip, whileyintan-gw does not. Everything works fine, except I need to occasionally click the "Refresh Hosts Keys" button. Otherwise, the networking between nodes is unreachable.

What happens when i click the "Refresh Hosts Keys" button, and how should i troubleshoot the issue?

Looking forward to your reply. Thank you.

image

Version

v0.24.0

What OS are you using?

Linux

Relevant log output

No response

Contributing guidelines

  • [X] Yes, I did.

liuweiGL avatar Apr 26 '24 23:04 liuweiGL

@liuweiGL , when clicking "Refresh Hosts Keys", it will trigger an event that the WG keys will be refreshed. And the event will be cascaded to client side and there will be client restart(to refresh the changes).

At first, it may be to find the root cause for the issue. When the issue happens again,

  1. Please have a check if the network unreachable issue happens on specific hosts or all the hosts each other.
  2. Please have a check the logs on server side with command docker logs netmaker
  3. Please have a check the logs on client side with command journalctl -u netclient > ./netclient.log

yabinma avatar Apr 29 '24 08:04 yabinma

Thank you for your reply. I will keep you updated

liuweiGL avatar Apr 29 '24 10:04 liuweiGL

My egress configurations: image image

When i access 172.19.0.0/16 from the 192.168.31.0/24, the netmaker server reports errors all the time:

{"time":"2024-04-30T02:14:58.274714922Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:15:58.274887289Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:16:58.274403311Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:17:58.274495202Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:18:58.274831076Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:19:58.274988418Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:20:58.274567037Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:21:58.274073556Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:22:58.274929475Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:23:58.274791097Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:24:58.27394898Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:25:58.274519803Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:26:58.274681452Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:27:58.274100879Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:28:58.274052047Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:29:58.273766173Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:30:58.274367741Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:31:58.274506574Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:32:58.274490309Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:33:58.274546262Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:34:58.274112409Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:35:58.274184689Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:36:58.274333125Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:37:58.273564644Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:38:58.274335675Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:39:58.274778466Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:40:58.273922369Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}

The aliyun-gw netclient:

Apr 27 07:26:39 gateway-default systemd[1]: Stopping Netclient Daemon...
Apr 27 07:26:39 gateway-default netclient[1203]: [netclient] 2024-04-27 07:26:39 checkin routine closed
Apr 27 07:26:39 gateway-default netclient[1203]: [netclient] 2024-04-27 07:26:39 closed endpoint detection
Apr 27 07:26:39 gateway-default systemd[1]: netclient.service: Deactivated successfully.
Apr 27 07:26:39 gateway-default systemd[1]: Stopped Netclient Daemon.
Apr 27 07:26:39 gateway-default systemd[1]: netclient.service: Consumed 1min 34.101s CPU time.
Apr 27 07:26:42 gateway-default systemd[1]: Starting Netclient Daemon...
Apr 27 07:26:59 gateway-default systemd[1]: Started Netclient Daemon.
Apr 27 07:26:59 gateway-default netclient[108886]: daemon called
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 Starting firewall...
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 iptables is supported
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding forwarding rule
Apr 27 07:26:59 gateway-default netclient[108886]: completed pull for server netmaker.eastcoal.club
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding addresses to netmaker interface
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 initialized endpoint detection on port 51821
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding addresses to netmaker interface
Apr 30 07:56:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:56:29.860094683+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:56:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:56:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:56:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:56:59.859049671+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:57:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:57:29.858288344+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:57:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:57:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:57:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:57:59.881952834+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:58:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:58:29.85369169+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:58:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:58:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:58:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:58:59.880765587+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:59:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:59:29.879677495+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:59:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:59:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:59:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:59:59.880666504+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:00:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:00:29.854193914+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:00:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:00:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:00:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:00:59.875503138+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:01:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:01:29.881955996+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:01:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:01:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:01:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:01:59.88269573+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:02:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:02:29.855246497+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:02:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:02:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:02:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:02:59.889906515+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:03:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:03:29.881813487+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:03:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:03:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:03:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:03:59.883502242+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:04:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:04:29.859572504+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:04:52 gateway-default netclient[108886]: [netclient] 2024-04-30 08:04:52 adding addresses to netmaker interface
Apr 30 08:06:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:06:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:06:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:06:59.851468004+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 503 Service Unavailable <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:07:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:07:29.851430857+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 503 Service Unavailable <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:07:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:07:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:07:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:07:59.879864985+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 502 Bad Gateway <html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:08:30 gateway-default netclient[108886]: completed pull for server netmaker.eastcoal.club

The yintan-gw node is openwrt os, i can not find any logs in /var/log/syslog

liuweiGL avatar Apr 30 '24 02:04 liuweiGL

The current state is that the network is blocked when the traffic is heavy otherwise the network is fine.

eg: Ten clients simultaneously sending 100MB files.

liuweiGL avatar Apr 30 '24 02:04 liuweiGL

@liuweiGL are you using self-hosted or SaaS version of netmaker?

abhishek9686 avatar Apr 30 '24 05:04 abhishek9686

I'm using self-hosted, both the client and server versions are v0.24.0

liuweiGL avatar Apr 30 '24 05:04 liuweiGL

I believe you are using nginx reverse proxy rather than the default offering caddy?

abhishek9686 avatar Apr 30 '24 05:04 abhishek9686

I install the netmaker in k8s cluster base on https://github.com/gravitl/netmaker-helm, the chart installed with values:

baseDomain: "netmaker.example.club" 

serviceAccount:
  # -- Specifies whether a service account should be created
  create: true
  # -- Annotations to add to the service account
  annotations: {}
  # -- Name of SA to use. If not set and create is true, a name is generated using the fullname template
  name: ""

ui:
  # -- how many UI replicas to create
  replicas: 1
  image:
    # -- The image repo to pull Netmaker image from
    repository: gravitl/netmaker-ui
    # -- Pull Policy for images
    pullPolicy: Always
    # -- Override the image tag to pull
    tag: "v0.24.0"

mq:
  # -- how many MQTT replicas to create
  replicas: 1
  username: netmaker
  password: *********
server:
  # -- number of netmaker server replicas to create
  replicas: 1
  image:
    # -- The image repo to pull Netmaker image from
    repository: gravitl/netmaker
    # -- Pull Policy for images
    pullPolicy: Always
    # -- Override the image tag to pull
    tag: "v0.24.0"
  RWX:
    storageClassName: "longhorn"
  storageSize: 1G
  masterKey: "netmaker"
  ee:
    licensekey: ""
    tenantId: ""
   # Duration of JWT token validity in seconds
  jwtDuration: 86400
  # Auto disable a user's connecteds clients bassed on JWT token expiration
  racAutoDisable: "true" 
   # "<azure-ad|github|google|oidc>"
  authProvider: ""  
   # "<client id of your oauth provider>"
  oAuthclientID: ""
   # "<client secret of your oauth provider>"
  oAuthClientSecret: ""
   # "https://dashboard.<netmaker base domain>"
  frontendURL: ""
  oidcIssuer: "" 
  turn:
    enabled: false

dns:
  enabled: false

setIpForwarding:
  enabled: true

service:
  # -- type for netmaker server services
  type: ClusterIP
  # -- port for API service
  restPort: 8081
  # -- port for MQTT service
  mqPort: 8082
  uiPort: 80

ingress:
  enabled: true
  tls: true
  className: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: zerossl-http
  hostPrefix:
    ui: "dashboard"
    broker: "broker"
    rest: "api"
db:
  type: "postgres"
  host: "postgresql.netmaker.svc.cluster.local"
  port: 5432
  username: netmaker
  # -- postgres pass to generate
  password: *******
  database: netmaker

postgresql-ha:
  enabled: false

liuweiGL avatar Apr 30 '24 05:04 liuweiGL

alright this is a installation on k8s. i see some errors logs related to your domain and I believe there are issues with the nginx setup

My egress configurations: image image

When i access 172.19.0.0/16 from the 192.168.31.0/24, the netmaker server reports errors all the time:

{"time":"2024-04-30T02:14:58.274714922Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:15:58.274887289Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:16:58.274403311Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:17:58.274495202Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:18:58.274831076Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:19:58.274988418Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:20:58.274567037Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:21:58.274073556Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:22:58.274929475Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:23:58.274791097Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:24:58.27394898Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:25:58.274519803Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:26:58.274681452Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:27:58.274100879Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:28:58.274052047Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:29:58.273766173Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:30:58.274367741Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:31:58.274506574Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:32:58.274490309Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:33:58.274546262Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:34:58.274112409Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:35:58.274184689Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:36:58.274333125Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:37:58.273564644Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:38:58.274335675Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:39:58.274778466Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:40:58.273922369Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}

The aliyun-gw netclient:

Apr 27 07:26:39 gateway-default systemd[1]: Stopping Netclient Daemon...
Apr 27 07:26:39 gateway-default netclient[1203]: [netclient] 2024-04-27 07:26:39 checkin routine closed
Apr 27 07:26:39 gateway-default netclient[1203]: [netclient] 2024-04-27 07:26:39 closed endpoint detection
Apr 27 07:26:39 gateway-default systemd[1]: netclient.service: Deactivated successfully.
Apr 27 07:26:39 gateway-default systemd[1]: Stopped Netclient Daemon.
Apr 27 07:26:39 gateway-default systemd[1]: netclient.service: Consumed 1min 34.101s CPU time.
Apr 27 07:26:42 gateway-default systemd[1]: Starting Netclient Daemon...
Apr 27 07:26:59 gateway-default systemd[1]: Started Netclient Daemon.
Apr 27 07:26:59 gateway-default netclient[108886]: daemon called
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 Starting firewall...
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 iptables is supported
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding forwarding rule
Apr 27 07:26:59 gateway-default netclient[108886]: completed pull for server netmaker.eastcoal.club
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding addresses to netmaker interface
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 initialized endpoint detection on port 51821
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding addresses to netmaker interface
Apr 30 07:56:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:56:29.860094683+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:56:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:56:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:56:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:56:59.859049671+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:57:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:57:29.858288344+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:57:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:57:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:57:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:57:59.881952834+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:58:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:58:29.85369169+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:58:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:58:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:58:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:58:59.880765587+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:59:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:59:29.879677495+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:59:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:59:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:59:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:59:59.880666504+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:00:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:00:29.854193914+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:00:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:00:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:00:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:00:59.875503138+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:01:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:01:29.881955996+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:01:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:01:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:01:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:01:59.88269573+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:02:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:02:29.855246497+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:02:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:02:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:02:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:02:59.889906515+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:03:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:03:29.881813487+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:03:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:03:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:03:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:03:59.883502242+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:04:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:04:29.859572504+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:04:52 gateway-default netclient[108886]: [netclient] 2024-04-30 08:04:52 adding addresses to netmaker interface
Apr 30 08:06:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:06:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:06:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:06:59.851468004+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 503 Service Unavailable <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:07:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:07:29.851430857+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 503 Service Unavailable <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:07:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:07:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:07:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:07:59.879864985+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 502 Bad Gateway <html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:08:30 gateway-default netclient[108886]: completed pull for server netmaker.eastcoal.club

The yintan-gw node is openwrt os, i can not find any logs in /var/log/syslog

abhishek9686 avatar Apr 30 '24 05:04 abhishek9686

alright this is a installation on k8s. i see some errors logs related to your domain and I believe there are issues with the nginx setup

My egress configurations: image image

When i access 172.19.0.0/16 from the 192.168.31.0/24, the netmaker server reports errors all the time:

{"time":"2024-04-30T02:14:58.274714922Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:15:58.274887289Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:16:58.274403311Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:17:58.274495202Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:18:58.274831076Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:19:58.274988418Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:20:58.274567037Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:21:58.274073556Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:22:58.274929475Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:23:58.274791097Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:24:58.27394898Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:25:58.274519803Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:26:58.274681452Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:27:58.274100879Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:28:58.274052047Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:29:58.273766173Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:30:58.274367741Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:31:58.274506574Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:32:58.274490309Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:33:58.274546262Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:34:58.274112409Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:35:58.274184689Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:36:58.274333125Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:37:58.273564644Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:38:58.274335675Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:39:58.274778466Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}
{"time":"2024-04-30T02:40:58.273922369Z","level":"ERROR","source":"handlers.go 90}","msg":"error getting host","id":"b9e3d6c2-6032-4a54-9569-42df81a0bcfc","error":"no result found"}

The aliyun-gw netclient:

Apr 27 07:26:39 gateway-default systemd[1]: Stopping Netclient Daemon...
Apr 27 07:26:39 gateway-default netclient[1203]: [netclient] 2024-04-27 07:26:39 checkin routine closed
Apr 27 07:26:39 gateway-default netclient[1203]: [netclient] 2024-04-27 07:26:39 closed endpoint detection
Apr 27 07:26:39 gateway-default systemd[1]: netclient.service: Deactivated successfully.
Apr 27 07:26:39 gateway-default systemd[1]: Stopped Netclient Daemon.
Apr 27 07:26:39 gateway-default systemd[1]: netclient.service: Consumed 1min 34.101s CPU time.
Apr 27 07:26:42 gateway-default systemd[1]: Starting Netclient Daemon...
Apr 27 07:26:59 gateway-default systemd[1]: Started Netclient Daemon.
Apr 27 07:26:59 gateway-default netclient[108886]: daemon called
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 Starting firewall...
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 iptables is supported
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding forwarding rule
Apr 27 07:26:59 gateway-default netclient[108886]: completed pull for server netmaker.eastcoal.club
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding addresses to netmaker interface
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 initialized endpoint detection on port 51821
Apr 27 07:26:59 gateway-default netclient[108886]: [netclient] 2024-04-27 07:26:59 adding addresses to netmaker interface
Apr 30 07:56:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:56:29.860094683+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:56:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:56:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:56:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:56:59.859049671+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:57:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:57:29.858288344+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:57:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:57:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:57:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:57:59.881952834+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:58:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:58:29.85369169+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:58:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:58:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:58:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:58:59.880765587+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:59:29 gateway-default netclient[108886]: {"time":"2024-04-30T07:59:29.879677495+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 07:59:59 gateway-default netclient[108886]: [netclient] 2024-04-30 07:59:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 07:59:59 gateway-default netclient[108886]: {"time":"2024-04-30T07:59:59.880666504+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:00:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:00:29.854193914+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:00:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:00:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:00:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:00:59.875503138+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:01:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:01:29.881955996+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:01:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:01:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:01:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:01:59.88269573+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:02:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:02:29.855246497+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:02:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:02:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:02:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:02:59.889906515+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:03:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:03:29.881813487+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:03:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:03:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:03:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:03:59.883502242+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:04:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:04:29.859572504+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"Post \"https://api.netmaker.eastcoal.club:443/api/hosts/adm/authenticate\": dial tcp 47.101.43.254:443: connect: connection refused"}
Apr 30 08:04:52 gateway-default netclient[108886]: [netclient] 2024-04-30 08:04:52 adding addresses to netmaker interface
Apr 30 08:06:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:06:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:06:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:06:59.851468004+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 503 Service Unavailable <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:07:29 gateway-default netclient[108886]: {"time":"2024-04-30T08:07:29.851430857+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 503 Service Unavailable <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:07:59 gateway-default netclient[108886]: [netclient] 2024-04-30 08:07:59 MQ client is not connected, using fallback checkin for server netmaker.eastcoal.club
Apr 30 08:07:59 gateway-default netclient[108886]: {"time":"2024-04-30T08:07:59.879864985+08:00","level":"ERROR","source":"mqhandlers.go 506}","msg":"pull failed","error":"failed to authenticate 502 Bad Gateway <html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
Apr 30 08:08:30 gateway-default netclient[108886]: completed pull for server netmaker.eastcoal.club

The yintan-gw node is openwrt os, i can not find any logs in /var/log/syslog

abhishek9686 avatar Apr 30 '24 05:04 abhishek9686

I'll reinstall it

liuweiGL avatar Apr 30 '24 06:04 liuweiGL

@liuweiGL any updates?

abhishek9686 avatar May 06 '24 15:05 abhishek9686

I've been a little busy lately, i'll try later

liuweiGL avatar May 07 '24 12:05 liuweiGL

Hi, I have made the following changes:

  1. I customized the helm chart, using NodePort instead of nginx to access mosquito
  2. The yintan-gw node has a sing-box proxy installed and uses tproxy for global proxy, now i have canceled forwarding of udp traffic
  3. Open port 51821 on both the yintan-gw and aliyun-gw nodes

Finally, I modified the systemd configuration to add -v 4 to print detailed logs, and I will sync up if there are any updates

liuweiGL avatar May 09 '24 06:05 liuweiGL

Hi, i have a new problem... The console shows healthy after my partner joins the network, but the network is not actually accessible.

Snipaste_2024-05-09_22-06-10 未命名文件

liuweiGL avatar May 09 '24 14:05 liuweiGL

How can i debug/test the peers connection? Or if i want to use the native wireguard client https://www.wireguard.com/install/ connect lx-office how should i configure it?

liuweiGL avatar May 09 '24 14:05 liuweiGL

How can i debug/test the peers connection? Or if I want to use the native wireguard client https://www.wireguard.com/install/ connect lx-office how should i configure it?

for devices behind a restrictive NAT, make sure you do port forwarding on the wireguard listen port, the firewall maybe blocking it. Since you are on the community version, relays are not available to overcome this.

for native wireguard client setup, remote access gw and you can create clients on it, which are basically raw wiregaurd configs. i would suggest for devices behind NAT on community version, use the remote access gw client configs for stable connection.

https://docs.netmaker.io/external-clients.html

abhishek9686 avatar May 09 '24 15:05 abhishek9686

Oh, and some information:

  1. yintan-gw and lx-office have the same network provider that is chinatelecom
  2. aliyun-gw is a vm service by alibaba cloud provider

liuweiGL avatar May 09 '24 15:05 liuweiGL

How can i debug/test the peers connection? Or if I want to use the native wireguard client https://www.wireguard.com/install/ connect lx-office how should i configure it?

for devices behind a restrictive NAT, make sure you do port forwarding on the wireguard listen port, the firewall maybe blocking it. Since you are on the community version, relays are not available to overcome this.

for native wireguard client setup, remote access gw and you can create clients on it, which are basically raw wiregaurd configs. i would suggest for devices behind NAT on community version, use the remote access gw client configs for stable connection.

https://docs.netmaker.io/external-clients.html

But yintan-gw can access lx-office , if it's a firewall problem, I don't know how to explain it

liuweiGL avatar May 09 '24 15:05 liuweiGL

The issue was resolved using Remote Access

liuweiGL avatar May 10 '24 02:05 liuweiGL

The issue was resolved using Remote Access

can we close this issue?

abhishek9686 avatar May 10 '24 02:05 abhishek9686

Thx

liuweiGL avatar May 10 '24 02:05 liuweiGL