Securing streams via config file

[DeanPDX]

I am having a hard time understanding how to secure my streams. I was able to get subject security working before adding in Jetstream with the following config:

# Authorization
authorization {
  default_permissions = {
    publish = "SANDBOX.*"
    subscribe = ["PUBLIC.>", "_INBOX.>"]
  }
  APIENV = {
    publish = ["api.>", "PUBLIC.>", "_INBOX.>"]
    subscribe = ["api.>", "PUBLIC.>", "_INBOX.>"]
  }
  INTEGRATIONENV = {
    publish = ["integration.>", "PUBLIC.>", "_INBOX.>"]
    subscribe = ["integration.>", "PUBLIC.>", "_INBOX.>"]
  }
  users = [
    {user: apiuser, password: "apipass" , permissions: $APIENV }
    {user: integration, password: "password" , permissions: $INTEGRATIONENV }
    {user: other, password: "otherpass"}
  ]
}

But as soon as I enable Jetstream, my security goes out the window and Jetstream just lets all messages be passed. For reference, here's part of a script I was running in an attempt to verify security was working how I wanted it to:

echo "Sending bad request (no credentials) \n"
nats request api.user.v1 '{"ID":{{.Count}}, "FullName": "Bad Request No Credentials!"}' -H route:create --count=1

... but it worked just fine without credentials. I found issue #2367 and it seemed similar to what I was experiencing. This comment specifically seemed relevant to my situation:

You should probably remove the authorization section you're essentially configuring 2 competing auth systems here, and specifically you have the backend user in both this section and the accounts section.

So, I got rid of my authorization section and instead went with this:

accounts = {
  # Regular application users, no jetstream access
  USER = {
    users = [
      {user: 'user', password: 'user'}
    ]
  }
  # API
  API = {
    users = [
        {user: apiuser, password: "apipass" }
    ]
    jetstream = enabled
    imports = [
      {stream: {subject: api.>, account: API}}
    ]
    exports = [
      {stream: api.>}
    ]
  }
  # integration service
  INTEGRATION = {
    users = [
      {user: integration, password: "password" }
    ]
    imports = [
      {stream: {subject: integration.>, account: INTEGRATION}}
    ]
    exports = [
      {stream: integration.>}
    ]
    jetstream = enabled
  }
}

I've also tried a lot of variations on this config to no avail. What am I missing here? How can I lock down stream access the same way I was locking down subjects? I can't seem to find a section of the docs that really goes very deep on this. And this comment made me feel even more confused:

I believe the account level users still accept authorization blocks too

So, we have an accounts block, which has a users block, which accepts an authorization block? I must be thinking of "accounts" in the wrong way. The upshot of what I want to accomplish is: I want to lock down streams by user, because I potentially want to run multiple clients' data through my messaging server. So, if NodeA belongs to ClientA it should only ever be able to see streams that I've designated for ClientA. Is my thinking about security completely wrong-headed on this? Is there a section in the docs that addresses this? Any help is appreciated!

For reference (in case it helps) here's my stream info:

$ nats stream info integration
Information for Stream integration created 2022-01-25T10:56:52-08:00

Configuration:

             Subjects: integration.>
     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

State:
             Messages: 12
                Bytes: 1.3 KiB
             FirstSeq: 1 @ 2022-01-25T18:57:30 UTC
              LastSeq: 12 @ 2022-01-25T19:11:56 UTC
     Active Consumers: 3

[derekcollison]

Yes of course you can lock down subjects and restrict access to those subjects. JetStream has nothing to do with that.

What context are you using for the nats cli?

What do you mean it worked just fine? The nats request properly returned a value from a subscriber you have? Again what nats context are you using when you make that request?

[DeanPDX]

Yes of course you can lock down subjects and restrict access to those subjects. JetStream has nothing to do with that.

Yes, I was able to lock down subjects before introducing JetStream. Introducing JetStream was when I started getting in to problem areas where I wanted to lock down specific streams and my thought was that perhaps it is related to the comment I mentioned suggesting that the other developer should probably remove the authorization section because there are 2 competing auth systems.

What context are you using for the nats cli?

I'm connecting to a server on my localhost currently, running on the official nats docker image:

$ nats context show local
NATS Configuration Context "local"

      Description: Localhost
      Server URLs: nats://127.0.0.1:4222
             Path: /home/deand/.config/nats/context/local.json

... and the context of /home/deand/.config/nats/context/local.json:

{
  "description": "Localhost",
  "url": "nats://127.0.0.1:4222",
  "token": "",
  "user": "",
  "password": "",
  "creds": "",
  "nkey": "",
  "cert": "",
  "key": "",
  "ca": "",
  "nsc": "",
  "jetstream_domain": "",
  "jetstream_api_prefix": "",
  "jetstream_event_prefix": ""
}

What do you mean it worked just fine? The nats request properly returned a value from a subscriber you have?

What I mean is that I was able to test security with different user/password combos. When I entered bad credentials that I expected would not be allowed to publish to a given subject, I got the following error:

# Script to test and generate error to validate that security is working how I want it to
echo "Sending bad request (no credentials) \n"
nats request api.user.v1 '{"ID":{{.Count}}, "FullName": "Bad Request No Credentials!"}' -H route:create --count=1

# Results
Sending bad request (no credentials)
nats: error: nats: Authorization Violation, try --help

With no change to the config other than Enabling JetStream (or without changing config file at all but using --js flag when starting docker image to enable JetStream), the results change to:

# Results with same config but jetstream enabled
Sending bad request (no credentials)
12:27:50 Sending request on "api.user.v1"
12:27:50 Received on "_INBOX.yRUmyte9SQpTP4NsHNQ4fJ.AZfgfByB" rtt 2.357955ms
{"ID":1001,"FullName":"Jack Johnson"}

Again what nats context are you using when you make that request?

Same answer as above. To connect using my apiuser credentials I'm using the following:

nc, err := nats.Connect("nats://apiuser:[email protected]:4222")

The response is coming from the following handler. It's not really relevant to this problem but here it is regardless (in case you're wondering what is generating the response the CLI is printing):

// Express interest in various subjects we care about
_, err = nc.Subscribe("api.user.*", func(m *nats.Msg) {
	// Determine API version from subject
	apiVersion := strings.TrimPrefix(m.Subject, "api.user.")
	// Get route header to determine where to route this in service.
	route := m.Header.Get("route")
	logger.Printf("Received a message with route header '%v' for version %v\n", route, apiVersion)
	res := nats.NewMsg(m.Reply)
	result, err := userservice.HandleRequest(apiVersion, route, m.Data)
	if err != nil {
		res.Header.Add("error", err.Error())
	}
	res.Data = result
	m.RespondMsg(res)
})

Again - advice is welcome. And if there's a section of the docs that could be relevant I would appreciate that. I can't seem to find anything that explains in detail how the config file is supposed to work and which keys control the aspects of nats/jetstream that I am trying to configure/test.

[derekcollison]

ok thanks for the additional information. If possible could you attach your complete server config here and we will take a closer look.

[DeanPDX]

Sure. Here is my config (it's very bare-bones at the moment):

nats-server.txt <- rename to nats-server.conf

I tried a bunch of other stuff but this was the one that was working until I tried firing it up alongside jetstream (starting docker container with js flag or just changing this config to enable jetstream).

[derekcollison]

Thanks will take a look.

[derekcollison]

ok so I took a look. First thing I did was remove the cluster designation, JetStream will not start with that config. I ran all the perm tests and they worked fine.

If you try to start the server with that config with -js it will fail. Could it be that it was failing in some strange way to start and it started a default one or something? The server should never have run with that config and the -js flag. It would complain about no server name, no cluster name, and no solicited routes which are all required for JetStream clustered.

[DeanPDX]

I just checked and I don't see any errors:

[1] 2022/01/26 20:27:39.558284 [INF] Starting nats-server
[1] 2022/01/26 20:27:39.558556 [INF]   Version:  2.6.6
[1] 2022/01/26 20:27:39.558568 [INF]   Git:      [878afad]
[1] 2022/01/26 20:27:39.558588 [INF]   Name:     NDOD5XGLGRYOCTF7XX5F5E4XSZZAQDDAXKI44B4QTIHJWGGN47ZMI6HJ
[1] 2022/01/26 20:27:39.558641 [INF]   Node:     gcb0ywgS
[1] 2022/01/26 20:27:39.558652 [INF]   ID:       NDOD5XGLGRYOCTF7XX5F5E4XSZZAQDDAXKI44B4QTIHJWGGN47ZMI6HJ
[1] 2022/01/26 20:27:39.563917 [INF] Starting JetStream
[1] 2022/01/26 20:27:39.565197 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[1] 2022/01/26 20:27:39.565240 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[1] 2022/01/26 20:27:39.565249 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[1] 2022/01/26 20:27:39.565256 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[1] 2022/01/26 20:27:39.565262 [INF]
[1] 2022/01/26 20:27:39.565269 [INF]          https://docs.nats.io/jetstream
[1] 2022/01/26 20:27:39.565275 [INF]
[1] 2022/01/26 20:27:39.565282 [INF] ---------------- JETSTREAM ----------------
[1] 2022/01/26 20:27:39.565302 [INF]   Max Memory:      5.64 GB
[1] 2022/01/26 20:27:39.565315 [INF]   Max Storage:     637.96 GB
[1] 2022/01/26 20:27:39.565325 [INF]   Store Directory: "/tmp/nats/jetstream"
[1] 2022/01/26 20:27:39.565337 [INF] -------------------------------------------
[1] 2022/01/26 20:27:39.567806 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2022/01/26 20:27:39.568456 [INF] Server is ready

I'll remove the cluster config for now and see if I can move forward without clustering for now. Hopefully that was the problem. I'll keep you posted if I find out more.

[derekcollison]

That run is not using a config file.. It will print out a config file when it is using one.

[derekcollison]

Also 2.7.1 is out, would suggest upgrading to that version. Will not affect what you are doing, but good to be current.