Can't start JetStream: could not create storage directory - mkdir /var/lib/ocis/nats: permission denied
I updated oCIS today and after I cleared all my pvcs for testing, the nats service refuses to come up:
2025-03-27T20:34:50Z INF Starting nats-server line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │ 2025-03-27T20:34:50Z INF Version: 2.10.22 line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats ││ 2025-03-27T20:34:50Z INF Git: [1b37ad3] line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │ 2025-03-27T20:34:50Z DBG Go build: go1.22.12 line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:45 service=nats │ 2025-03-27T20:34:50Z INF Cluster: ocis-cluster line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │ 2025-03-27T20:34:50Z INF Name: NB2SWWUZWZ6FGBJQNBPIQUKKEAO6GDJPYQ3KDLJOUN6VQ5W463LRYIYR line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │ 2025-03-27T20:34:50Z INF Node: 520V1e5t line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │ 2025-03-27T20:34:50Z INF ID: NB2SWWUZWZ6FGBJQNBPIQUKKEAO6GDJPYQ3KDLJOUN6VQ5W463LRYIYR line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │ 2025-03-27T20:34:50Z DBG Created system account: "$SYS" line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:45 service=nats │ 2025-03-27T20:34:50Z INF Starting JetStream line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │ 2025-03-27T20:34:50Z DBG JetStream creating dynamic configuration - 35.28 GB memory, 1.00 TB disk line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:45 service=nats │ 2025-03-27T20:34:50Z FTL Can't start JetStream: could not create storage directory - mkdir /var/lib/ocis/nats: permission denied line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:33 service=nats │ stream closed EOF for ocis/nats-75c6fdddfc-fgj8j (nats)
Hm - I could workaround it with chownInitContainer: true but that hadn't been necessary in this cluster before, so I'm a little confused on what changed here.
Can't start JetStream: could not create storage directory - mkdir /var/lib/ocis/nats: permission denied
I'm not sure what the oCIS chart should do about it.
chownInitContainer: true
is only there for CSI drivers that do not use the fsGroup from the securityContext, see also: https://github.com/owncloud/ocis- charts/blob/9614fb4cbc9f47e756dffcd28a217369fb12bb5f/charts/ocis/values.yaml#L3049-L3053
Furthermore I highly recommend you to not use the builtin NATS except for quick testing setups. If it is a testing setup only, you could just start from scratch and see if the issue persists.
If it is a testing setup only, you could just start from scratch and see if the issue persists.
Was it reproducible?
I will try to reproduce this or next week. The cluster wasn't ... operable ... for some time 😅