nats-server icon indicating copy to clipboard operation
nats-server copied to clipboard

Published 20 hours ago • verified publisher icon nats-io

how to enabled JetStream for account

Open rickiey opened this issue 3 years ago • 7 comments

$ nats --user=root --password=aaaaaa account info

Connection Information:

               Client ID: 17
               Client IP: 127.0.0.1
                     RTT: 152.236µs
       Headers Supported: true
         Maximum Payload: 1.0 MiB
       Connected Cluster: nc
           Connected URL: nats://127.0.0.1:4222
       Connected Address: 127.0.0.1:4222
     Connected Server ID: NDCVXNEA6YCSXRQEBE62XKMBDPMADOANPQH45TBWF26GVFBCT2XTO5DU
   Connected Server Name: nats-server-150

JetStream Account Information:

   Could not obtain account information: JetStream not enabled for account
  • my config:
server_name=nats-server-150

listen: "0.0.0.0:4222"

jetstream {
   store_dir=/var/lib/nats/storage
   max_mem: 30Gb
   max_file: 60Gb
}

accounts: {
    USERS: {
        jetstream: enable
        users: [
            {user: root, password: aaaaaa}
        ]
    },
    SYS: {
        users: [
            {user: admin, password: aaaaaa}
           ]
    },
}
system_account: SYS

cluster {
  # host/port for inbound route connections from other server
  name: nc
  listen: "192.168.55.150:4244"

  # Authorization for route connections
  # Other server can connect if they supply the credentials listed here
  # This server will connect to discovered routes using this user
  authorization {
    user: root
    password: aaaaaa
    timeout: "0.5"
  }

  connect_retries: 5

  # This server establishes routes with these server.
  # This server solicits new routes and Routes are actively solicited and connected to from this server.
  # Other servers can connect to us if they supply the correct credentials
  # in their routes definitions from above.
  routes: [
    nats-route://root:[email protected]:4244
    nats-route://root:[email protected]:4244
  ]
}

rickiey avatar Jul 22 '21 07:07 rickiey

I didn't see anything about this. Can I give a completed JetStream Cluster configuration document? Your document is really chaotic.

rickiey avatar Jul 22 '21 08:07 rickiey

Your configuration is correct and enables JetStream, maybe something was logged on the server to indicate a problem at startup?

ripienaar avatar Jul 22 '21 09:07 ripienaar

I've been experiencing something similar. I have the following configuration:

debug = true
trace = true
server_name = nats.myserver

port = 4222
monitor_port = 8222

jetstream = {
  store_dir = "/data/nats-server/"
  max_memory_store = 52428800
  max_file_store = 5368709120
}

# Authorization
authorization = {
  timeout = 3
  ADMIN = {
    publish = ">"
    subscribe = ">"
  }
  USER = {
    publish = "SANDBOX.*"
    subscribe = ["PUBLIC.>", "_INBOX.>", "someTopic1.user.*."]
  }
  users = [
    {user: $ADMIN_USER,  password: $ADMIN_USER, permissions: $ADMIN}
    {user: 'backend', password: 'backend', permissions: $ADMIN}
    {user: 'myservice', password: 'myservice', permissions: $ADMIN}
    {user: 'user', password: 'user', permissions: $USER}
  ]
}

# Accounts and stream config
accounts = {
  # Regular application users, no jetstream access
  USER = {
    users = [
      {user: 'user', password: 'user'}
    ]
    imports = [
      {stream: {subject: someTopic1.user.*, account: NOTIFICATION}}
    ]
  }
  # Service
  NOTIFICATION = {
    users = [
      {user: 'myservice', password: 'myservice'}
      {user: 'backend', password: 'backend'}
    ]
    jetstream = {
      max_mem: 24M
      max_file: 1G
      max_streams: 5
      max_consumers: 5
    }
    exports = [
      {stream: someTopic1.>}
    ]
  }
  ADMIN = {
    users = [
      {user: $ADMIN_USER, password: $ADMIN_USER}
    ]
    imports = [
      {stream: {subject: someTopic1.>, account: NOTIFICATION}}
    ]
    jetstream = enabled
  }
  # System
  SYS = {
    users = [
      {user: 'sys', password: $SYS}
    ]
  }
}
no_auth_user = user
system_account = SYS

websocket = {
    port = 4223
    no_tls = true
}

I start nats in one container, then connect to it from a second container and get this error:

[1] 2021/08/30 13:21:04.791624 [INF] Starting nats-server
[1] 2021/08/30 13:21:04.791653 [INF]   Version:  2.4.0
[1] 2021/08/30 13:21:04.791656 [INF]   Git:      [e49eb66]
[1] 2021/08/30 13:21:04.791658 [DBG]   Go build: go1.16.7
[1] 2021/08/30 13:21:04.791659 [INF]   Name:     nats.myserver
[1] 2021/08/30 13:21:04.791663 [INF]   Node:     QQNpzpjc
[1] 2021/08/30 13:21:04.791665 [INF]   ID:       NBLT6EMTRPWV72GDWVA2A3RBEJMPDETFLP7WWTSWDVTPRBIFEKE4MUPS
[1] 2021/08/30 13:21:04.791668 [WRN] Plaintext passwords detected, use nkeys or bcrypt
[1] 2021/08/30 13:21:04.791670 [INF] Using configuration file: /etc/nats/nats-server.conf
[1] 2021/08/30 13:21:04.792107 [INF] Starting JetStream
[1] 2021/08/30 13:21:04.792226 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[1] 2021/08/30 13:21:04.792230 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[1] 2021/08/30 13:21:04.792232 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[1] 2021/08/30 13:21:04.792234 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[1] 2021/08/30 13:21:04.792236 [INF] 
[1] 2021/08/30 13:21:04.792238 [INF]          https://docs.nats.io/jetstream
[1] 2021/08/30 13:21:04.792240 [INF] 
[1] 2021/08/30 13:21:04.792242 [INF] ---------------- JETSTREAM ----------------
[1] 2021/08/30 13:21:04.792249 [INF]   Max Memory:      50.00 MB
[1] 2021/08/30 13:21:04.792252 [INF]   Max Storage:     5.00 GB
[1] 2021/08/30 13:21:04.792254 [INF]   Store Directory: "/data/nats-server/jetstream"
[1] 2021/08/30 13:21:04.792256 [INF] -------------------------------------------
[1] 2021/08/30 13:21:04.792341 [DBG]   Exports:
[1] 2021/08/30 13:21:04.792345 [DBG]      $JS.API.>
[1] 2021/08/30 13:21:04.792370 [DBG] Enabled JetStream for account "ADMIN"
[1] 2021/08/30 13:21:04.792375 [DBG]   Max Memory:      -1 B
[1] 2021/08/30 13:21:04.792377 [DBG]   Max Storage:     -1 B
[1] 2021/08/30 13:21:04.792498 [DBG] JetStream state for account "ADMIN" recovered
[1] 2021/08/30 13:21:04.792519 [DBG] Enabled JetStream for account "NOTIFICATION"
[1] 2021/08/30 13:21:04.792524 [DBG]   Max Memory:      22.89 MB
[1] 2021/08/30 13:21:04.792528 [DBG]   Max Storage:     953.67 MB
[1] 2021/08/30 13:21:04.792650 [DBG] JetStream state for account "NOTIFICATION" recovered
[1] 2021/08/30 13:21:04.793673 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2021/08/30 13:21:04.794040 [INF] Listening for websocket clients on ws://0.0.0.0:4223
[1] 2021/08/30 13:21:04.794045 [WRN] Websocket not configured with TLS. DO NOT USE IN PRODUCTION!
[1] 2021/08/30 13:21:04.794050 [DBG] Get non local IPs for "0.0.0.0"
[1] 2021/08/30 13:21:04.883136 [DBG]   ip=192.168.240.2
[1] 2021/08/30 13:21:04.883196 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2021/08/30 13:21:04.883205 [DBG] Get non local IPs for "0.0.0.0"
[1] 2021/08/30 13:21:04.907550 [DBG]   ip=192.168.240.2
[1] 2021/08/30 13:21:04.907564 [INF] Server is ready
[1] 2021/08/30 13:21:08.206993 [DBG] 192.168.240.4:40935 - cid:7 - Client connection created
[1] 2021/08/30 13:21:08.207326 [DBG] 192.168.240.4:40935 - cid:7 - Client connection closed: Client Closed
[1] 2021/08/30 13:21:08.343782 [DBG] 192.168.240.4:54286 - cid:8 - Client connection created
[1] 2021/08/30 13:21:08.344134 [TRC] 192.168.240.4:54286 - cid:8 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"backend","pass":"[REDACTED]","tls_required":false,"name":"backend","lang":"go","version":"1.11.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
[1] 2021/08/30 13:21:08.344252 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <<- [PING]
[1] 2021/08/30 13:21:08.344258 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - ->> [PONG]
[1] 2021/08/30 13:21:08.344411 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <<- [SUB _INBOX.RCZ5h4JPVT4KS5k8r9zKuB.*  1]
[1] 2021/08/30 13:21:08.344430 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <<- [PUB $JS.API.INFO _INBOX.RCZ5h4JPVT4KS5k8r9zKuB.8E6jNhjp 0]
[1] 2021/08/30 13:21:08.344435 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <<- MSG_PAYLOAD: [""]
[1] 2021/08/30 13:21:08.344790 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - ->> [MSG _INBOX.RCZ5h4JPVT4KS5k8r9zKuB.8E6jNhjp 1 145]
[1] 2021/08/30 13:21:08.344814 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  579]
[1] 2021/08/30 13:21:08.344836 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"xeFl4pV2T0KFl5y0yE2w34\",\"timestamp\":\"2021-08-30T13:21:08.344674154Z\",\"server\":\"nats.myserver\",\"client\":{\"start\":\"2021-08-30T13:21:08.343739133Z\",\"host\":\"192.168.240.4\",\"id\":8,\"acc\":\"$G\",\"user\":\"backend\",\"name\":\"backend\",\"lang\":\"go\",\"ver\":\"1.11.0\",\"rtt\":403272,\"server\":\"nats.myserver\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.INFO\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.account_info_response\\\",\\\"error\\\":{\\\"code\\\":503,\\\"err_code\\\":10039,\\\"description\\\":\\\"JetStream not enabled for account\\\"}}\"}"]
[1] 2021/08/30 13:21:08.348156 [DBG] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - Client connection closed: Client Closed
[1] 2021/08/30 13:21:08.348179 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <-> [DELSUB 1]
[1] 2021/08/30 13:21:11.469922 [DBG] 192.168.240.4:54306 - cid:10 - Client connection created
[1] 2021/08/30 13:21:11.470305 [TRC] 192.168.240.4:54306 - cid:10 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"backend","pass":"[REDACTED]","tls_required":false,"name":"backend","lang":"go","version":"1.11.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
[1] 2021/08/30 13:21:11.470384 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <<- [PING]
[1] 2021/08/30 13:21:11.470390 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - ->> [PONG]
[1] 2021/08/30 13:21:11.470554 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <<- [SUB _INBOX.hjR8szP0693GIzb0NyDOfK.*  1]
[1] 2021/08/30 13:21:11.470582 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <<- [PUB $JS.API.INFO _INBOX.hjR8szP0693GIzb0NyDOfK.C9whJU49 0]
[1] 2021/08/30 13:21:11.470592 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <<- MSG_PAYLOAD: [""]
[1] 2021/08/30 13:21:11.470771 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - ->> [MSG _INBOX.hjR8szP0693GIzb0NyDOfK.C9whJU49 1 145]
[1] 2021/08/30 13:21:11.470785 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  580]
[1] 2021/08/30 13:21:11.470803 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"xeFl4pV2T0KFl5y0yE2w5j\",\"timestamp\":\"2021-08-30T13:21:11.470721369Z\",\"server\":\"nats.myserver\",\"client\":{\"start\":\"2021-08-30T13:21:11.469881976Z\",\"host\":\"192.168.240.4\",\"id\":10,\"acc\":\"$G\",\"user\":\"backend\",\"name\":\"backend\",\"lang\":\"go\",\"ver\":\"1.11.0\",\"rtt\":434441,\"server\":\"nats.myserver\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.INFO\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.account_info_response\\\",\\\"error\\\":{\\\"code\\\":503,\\\"err_code\\\":10039,\\\"description\\\":\\\"JetStream not enabled for account\\\"}}\"}"]
[1] 2021/08/30 13:21:11.474161 [DBG] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - Client connection closed: Client Closed
[1] 2021/08/30 13:21:11.474185 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <-> [DELSUB 1]

After this, the service periodically attempts to reconnect but keeps getting the JetStream not enabled for account error over and over.

Interestingly, this doesn't happen all the time. I had to restart the whole setup several times to reproduce this (for 80+% of the attempts it was working as expected). Invoking nats-server --signal reload multiple times after startup seems to fix it, but ofc it's not a long-term solution.

Any ideas on whether my config is incorrect, or if there is some race condition when loading the config (maybe some authorization + accounts inconsistency)?

lorenzodonini avatar Aug 30 '21 13:08 lorenzodonini

You should probably remove the authorization section you're essentially configuring 2 competing auth systems here, and specifically you have the backend user in both this section and the accounts section.

ripienaar avatar Aug 30 '21 14:08 ripienaar

@ripienaar I thought it would still be possible to combine authorization-like permission maps with Jetstream. Thanks for the clarification, docs weren't clear about that. Error seems to be gone 😃

lorenzodonini avatar Aug 31 '21 07:08 lorenzodonini

I believe the account level users still accept authorization blocks too

ripienaar avatar Aug 31 '21 07:08 ripienaar

Maybe try the nsc tool to add a new nats account and user and set jetstream (js) permissions? Worked for me at least...

Like this:

nsc add account MYNATSACCOUNT nsc add user MYNATSUSER export NATS_CA=/path/to/selfsigned/cluster/ca.cer nsc edit account --name MYNATSACCOUNT --js-mem-storeage -1 --js-disk-storage -1 --js-streams -1 --js-consumer -1 nsc push -a MYNATSACCOUNT -u nats://natsclusterurl.com

and finally use the credentials file output in [/unixuserhome/ or root] /.nkeys/creds/... to connect with nats-cluster

nats ..... --creds /path/to/user.creds --tlsca=/path/to/selfsigned/cluster/ca.cer

Zeg0 avatar Apr 20 '22 16:04 Zeg0

Seems you'll have to add --js-mem-storage -1 and/or --js-disk-storage -1 when you generate the account JWT.

lionello avatar Jun 12 '23 23:06 lionello