nats-server
nats-server copied to clipboard
nats-io
Allow Responses user permission not transiting leaf connection causing blocked service replies
In a scenario where the Hub user of a Leaf Node connection has defined user-level permissions, the allow responses subject permission does not work. A service responder at Leaf (mapped account) publishes a reply message but it is not received by the requester at Hub. If publish permission _INBOX.> is added to the Hub user, then the reply message is received by the requester at Hub.
Expectation is that Allow Response (as well as subject pub and sub permissions) would transit the connection user-account map.
nats-server: v2.7.5-beta.3
Here are example permission sets (OK and NOT OK):
OK (Request received at Leaf, Reply received at Hub)
+------------------------------------------------------------------------------------+
| User |
+-------------------------+----------------------------------------------------------+
| Name | edge-469R7A9 |
| User ID | UA2NB6JSKDAX4F6OKD23GA42YHBHLTGQ4XECBWI3QKEY5YNDRAZZO4HM |
| Issuer ID | AD5FKXQPZYFD6MI4WF23SMWAFWTAFFU44XDIH2N222XY5HUBL35WGH3L |
| Issued | 2022-03-23 23:02:20 UTC |
| Expires | |
| Bearer Token | No |
+-------------------------+----------------------------------------------------------+
| Pub Allow | _INBOX.> |
| Sub Allow | basestation.469R7A9.updater.> |
| Max Responses | 1 |
| Response Permission TTL | 5s |
+-------------------------+----------------------------------------------------------+
| Max Msg Payload | Unlimited |
| Max Data | Unlimited |
| Max Subs | Unlimited |
| Network Src | Any |
| Time | Any |
+-------------------------+----------------------------------------------------------+
NOT OK (Request received at Leaf, Reply not received at Hub, Hub requester times out)
+------------------------------------------------------------------------------------+
| User |
+-------------------------+----------------------------------------------------------+
| Name | edge-469R7A9 |
| User ID | UA2NB6JSKDAX4F6OKD23GA42YHBHLTGQ4XECBWI3QKEY5YNDRAZZO4HM |
| Issuer ID | AD5FKXQPZYFD6MI4WF23SMWAFWTAFFU44XDIH2N222XY5HUBL35WGH3L |
| Issued | 2022-03-23 23:13:17 UTC |
| Expires | |
| Bearer Token | No |
+-------------------------+----------------------------------------------------------+
| Sub Allow | basestation.469R7A9.updater.> |
| Max Responses | 1 |
| Response Permission TTL | 5s |
+-------------------------+----------------------------------------------------------+
| Max Msg Payload | Unlimited |
| Max Data | Unlimited |
| Max Subs | Unlimited |
| Network Src | Any |
| Time | Any |
+-------------------------+----------------------------------------------------------+
OK (Request received at Leaf, Reply received at Hub)
+---------------------------------------------------------------------------------+
| User |
+----------------------+----------------------------------------------------------+
| Name | edge-469R7A9 |
| User ID | UA2NB6JSKDAX4F6OKD23GA42YHBHLTGQ4XECBWI3QKEY5YNDRAZZO4HM |
| Issuer ID | AD5FKXQPZYFD6MI4WF23SMWAFWTAFFU44XDIH2N222XY5HUBL35WGH3L |
| Issued | 2022-03-23 23:16:55 UTC |
| Expires | |
| Bearer Token | No |
+----------------------+----------------------------------------------------------+
| Pub Allow | _INBOX.> |
| Sub Allow | basestation.469R7A9.updater.> |
| Response Permissions | Not Set |
+----------------------+----------------------------------------------------------+
| Max Msg Payload | Unlimited |
| Max Data | Unlimited |
| Max Subs | Unlimited |
| Network Src | Any |
| Time | Any |
+----------------------+----------------------------------------------------------+
Are we localizing only pub and sub permissions here causing the Leaf-side proxy to drop reply propagation to hub as it sees no merged permission (second NOT OK case above)?
https://github.com/tbeets/nats-server/blob/9a2da9ed8c04992baa66a558069f09d0b717a533/server/leafnode.go#L1442
@derekcollison For use-case orientation, this is smart API routing from cloud to fleet of edge nodes. https://github.com/ConnectEverything/ngs-edge-guide/blob/main/static/EdgeScenario.png
I have a "feeling" that I see a similar problem in a 2.7.4 r2 cluster B using a leafnode connection to a 2.7.4 r3 cluster A.
We have clients connecting to cluster B that consumes services (req-repl) in A. In 2.6.6 this connection was working. Now in 2.7.4 the messages seems to just disappear.
What did you do @tbeets to get it working? What user did you change, the requesting user in your hub or the user used for the leafnode connection?
@derekcollison I am experiencing this issue now.
I have a set-up to reproduce it: https://github.com/MikaelElkiaer/nats-examples/tree/main/leafnode-reply-allow_responses
Only way to get around this is to either:
- set
allow_responses: false- implicitly falling back topublish: { allow: > }, or - explicitly setting
publish: { allow: _R_.> }.
If hub user has explicit publish permissions, allow_responses does not provide a temporary override. My guess is that allow_responses works as usual local to hub but does not cause a temporary subscription to transit to leaf to attract the remote response. This is true in main and dev.
Workaround is to add the REPLY subject wildcard to the publish permissions at hub. This could be _R_.> if the requester traversed an account boundary at hub.
@derekcollison
I would have allow_responses on the creds used by the responders connected to the leaf and do what @tbeets recommends in terms of the user for the leafnode connection itself.