start_dev: error setting up proxy commands in container

[Nafalgar]

What happened? Starting the dev pipeline results in creating the pod successfully and starting the proxies, however the process exits with a 'error writing command' message:

info Using namespace '<namespace redacted>'
info Using kube context '<context redacted>'
deploy:bot Skipping deployment bot
dev:bot Waiting for pod to become ready...
dev:bot Selected botservice-deployment-devspace-84447cd8f-pchsf:botservice (pod:container)
dev:bot open  Opening 'http://localhost:3978' as soon as application will be started
dev:bot sync  Sync started on: ./ <-> ./
dev:bot sync  Waiting for initial sync to complete
dev:bot ssh   Port forwarding started on: 10997 -> 8022
dev:bot ports Port forwarding started on: 5186 -> 5186, 3978 -> 3978
dev:bot proxy Port forwarding started on: 10210 <- 10567
dev:bot ssh   Use 'ssh bot.bot.devspace' to connect via SSH
dev:bot sync  Initial sync completed
start_dev: error setting up proxy commands in container:  Error: error writing command '/usr/local/bin/devspace': open /usr/local/bin/devspace: read-only file system
Usage:
  devspacehelper proxy-commands configure [flags]

Flags:
      --commands strings     Commands to overwrite
      --git-credentials      If git credentials should get configured
  -h, --help                 help for configure
      --private-key string   Private key to use
      --public-key string    Public key to use
      --working-dir string   Working dir to use

error writing command '/usr/local/bin/devspace': open /usr/local/bin/devspace: read-only file system command terminated with exit code 1
fatal exit status 1

What did you expect to happen instead? The pipeline to execute successfully and set up the dev environment as intended

How can we reproduce the bug? (as minimally and precisely as possible)

My devspace.yaml:

version: v2beta1
name: bot

# This is a list of `pipelines` that DevSpace can execute (you can define your own)
pipelines:
  # This is the pipeline for the main command: `devspace dev` (or `devspace run-pipeline dev`)
  dev:
    run: |-
      run_dependencies --all       # 1. Deploy any projects this project needs (see "dependencies")
      create_deployments --all     # 2. Deploy Helm charts and manifests specfied as "deployments"
      start_dev bot                # 3. Start dev mode "app" (see "dev" section)
  # You can run this pipeline via `devspace deploy` (or `devspace run-pipeline deploy`)
  deploy:
    run: |-
      run_dependencies --all                            # 1. Deploy any projects this project needs (see "dependencies")
      build_images --all -t $(git describe --always)    # 2. Build, tag (git commit hash) and push all images (see "images")
      create_deployments --all                          # 3. Deploy Helm charts and manifests specfied as "deployments"

# This is a list of `images` that DevSpace can build for this project
# We recommend to skip image building during development (devspace dev) as much as possible
images:
  app:
    image: <redacted>
    dockerfile: <redacted>
    context: <redacted>

# This is a list of `deployments` that DevSpace can create for this project
deployments:
  bot:
    # This deployment uses `helm` but you can also define `kubectl` deployments or kustomizations
    helm:
      # We are deploying this project with the Helm chart you provided
      chart:
        name: ../Kubernetes/Helm/bot
      # Under `values` we can define the values for this Helm chart used during `helm install/upgrade`
      # You may also use `valuesFiles` to load values from files, e.g. valuesFiles: ["values.yaml"]
      valuesFiles: ["../Kubernetes/Helm/bot/values-test.yaml"]

# This is a list of `dev` containers that are based on the containers created by your deployments
dev:
  bot:
    # Search for the container that runs this image
    imageSelector: <redacted>
    # Replace the container image with this dev-optimized image (allows to skip image building during development)
    devImage: ghcr.io/loft-sh/devspace-containers/dotnet:6.0-alpine
    # Sync files between the local filesystem and the development container
    sync:
      - path: ./
        uploadExcludeFile: .dockerignore
    # Open a terminal and use the following command to start it
    terminal:
      command: ./devspace_start.sh
    # Inject a lightweight SSH server into the container (so your IDE can connect to the remote dev env)
    ssh:
      enabled: true
    # Make the following commands from my local machine available inside the dev container
    proxyCommands:
      - command: devspace
      - command: kubectl
      - command: helm
      - command: git
    # Forward the following ports to be able access your application via localhost
    ports:
      - port: "5186"
      - port: "3978"
    # Open the following URLs once they return an HTTP status code other than 502 or 503
    open:
      - url: http://localhost:3978

# Use the `commands` section to define repeatable dev workflows for this project 
commands:
  migrate-db:
    command: |-
      echo 'This is a cross-platform, shared command that can be used to codify any kind of dev task.'
      echo 'Anyone using this project can invoke it via "devspace run migrate-db"'

# Define dependencies to other projects with a devspace.yaml
# dependencies:
#   api:
#     git: https://...  # Git-based dependencies
#     tag: v1.0.0
#   ui:
#     path: ./ui        # Path-based dependencies (for monorepos)

Local Environment:

• DevSpace Version:devspace version 6.0.0-beta.6
• Operating System: windows
• ARCH of the OS: AMD64 Kubernetes Cluster:
• Cloud Provider: azure
• Kubernetes Version: Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"42a9a90338d705a1650fb68b7891f84b62adb0b0", GitTreeState:"clean", BuildDate:"2022-06-15T04:25:21Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Anything else we need to know? The devspace, helm, git, kubectl commands are all present in the pod in the intended path ('/usr/local/bin/') and are executable without errors, it does not have write permission and I can not use chmod to assign these

/kind bug

[tukobadnyanoba]

Hi @Nafalgar, I checked above yaml with devspace-quickstart-golang project and KinD(v1.22.5) cluster running on windows, it worked fine there. So, the different thing here is Azure here. What I suspect is readOnlyRootFilesystem flag set in the container. Could you please describe the pod?

[Nafalgar]

Thanks for the reply, the output of kubectl describe is as follows:

Name:         botservice-deployment-devspace-54cbd465dc-8zkzl
Namespace:    test1
Priority:     0
Node:         aks-agentpool-23858973-vmss000002/10.5.3.161
Start Time:   Wed, 13 Jul 2022 11:39:10 +0200
Labels:       aadpodidbinding=keyvaultidentity
              app=botservice
              devspace.sh/replaced=true
              pod-template-hash=54cbd465dc
Annotations:  checksum/configGeneral: 9f9c6f00e78e2159455bb41ee4398fc6d30e7b630b910aa106c43fcd77935646
              checksum/configSpecific: a6447a71147d4838578dab8056b9d43310bf79b4c09d10ba6db57c4ae6937cca
              devspace.sh/container: botservice
              devspace.sh/imageSelector: <redacted>
Status:       Running
IP:           10.5.4.131
IPs:
  IP:           10.5.4.131
Controlled By:  ReplicaSet/botservice-deployment-devspace-54cbd465dc
Containers:
  botservice:
    Container ID:  containerd://5cc141e4886fc7083ccf35c1f5290dc7f76166c7a2a4b5ff087ba702621cd184
    Image:         ghcr.io/loft-sh/devspace-containers/dotnet:6.0-alpine
    Image ID:      ghcr.io/loft-sh/devspace-containers/dotnet@sha256:5212b7bec4c9c7f3911a0ba50be089ff880a2af53a51d96e1597f8006cab07c0
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      sleep
      1000000000
    State:          Running
      Started:      Wed, 13 Jul 2022 11:39:11 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  300Mi
    Requests:
      cpu:        66m
      memory:     150Mi
    Environment:  <none>
    Mounts:
      /app/config from general-config-volume (rw)
      /app/config/specific from specific-config-volume (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5jmg6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  general-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      bot-general-config
    Optional:  false
  specific-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      bot-specific-config
    Optional:  false
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-5jmg6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

It doesn't appear to be using the readOnlyRootFilesystem flag.

[FabianKramm]

@Nafalgar do you need the proxy commands in your case? Seems like the folder /usr/local/bin is read only for some reason and cannot be written by DevSpace

[Nafalgar]

I don't think we need them, but it fails because it tries to use them. I already tried removing it from the config file, but that throws errors as well. Is there a way to disable them that I am overlooking?

[FabianKramm]

@Nafalgar whats the error when you disable those? It should just be erasing the dev.bot.proxyCommands option

[alexandradragodan]

Hello!

Thanks for opening this issue and helping out! Have you managed to disable the proxyCommands in dev.bot.proxyCommands?

We'd appreciate it if you could provide us with an update on how's that going on your side. Thanks.

[hwo411]

Hello, I also ran into this issue when I was trying to setup devspace for our apps. It seems that the issue happens when container is not run under root.

Of course, not proxying the commands helps, I wonder if it can be resolved without running container as root user (e.g., by just executing commands as root, but not running the entire container as root user).