Pebble Error `cannot create directory: permission denied` when pushing file to container on Juju 3.5.1

[NohaIhab]

Bug Description

On Juju 3.5.1

This issue is blocking the deployment and the CI for some CKF charms. As reported in this Juju 3.5.1 bug, our charms are not able to push files with pebble to the workload container with the error:

ops.pebble.PathError: permission-denied - cannot create directory: mkdir <dir_name>: permission denied

where <dir_name> is different for every charm, depending on what file it is attempting to push.

It looks like pebble does not have permissions to write files to workload containers.

This bug is known to affect the following charms, where the CI is failing in the corresponding PRs:

• jupyter-ui PR
• kfp-profile-controller PR
• kfp-persistence PR (same PR as kfp-profile-controller)

In our CI, we have Juju pinned to 3.5/stable, and for the Juju snap 3.5/stable is pinned to 3.5.1:

snap info juju | grep 3.5/stable
tracking:     3.5/stable
  3.5/stable:    3.5.1             2024-05-30 (27225)  99MB -

i.e. we have 3.5.1 in all of our CI at the moment of creating this issue.

The CI was seen to fail in main for the listed charms. Similarly, we expect this to break in the track/<version> branches.

To Reproduce

1. Install Juju 3.5/stable
2. Install microk8s 1.25-strict/stable
3. Bootstrap controller and add model
4. Deploy one of the charms mentioned in the description from latest/edge
5. Check the debug logs with juju debug-log --replay

Environment

juju 3.5.1 charms channel: latest/edge

Relevant Log Output

jupyter-ui: https://pastebin.canonical.com/p/2XJmKhdKrn/ kfp-persistence: https://pastebin.canonical.com/p/5zT8qk6bBS/ kfp-profile-controller: https://pastebin.canonical.com/p/vv9MwYN9y5/

Additional Context

No response