Self-hosted v3 function app fails to run in K8S pod if filesystem is readonly

[springcomp]

I’m hosting the Functions runtime in a dotnet Core 3.1 app to run as a Docker image inside a Kubernetes cluster. However, the function app fails early on startup with the following stack trace:

System.IO.IOException: Read-only file system
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at System.IO.Abstractions.DirectoryWrapper.CreateDirectory(String path)
   at Microsoft.Azure.WebJobs.Script.FileUtility.EnsureDirectoryExists(String path) in /src/azure-functions-host/src/WebJobs.Script/Extensions/FileUtility.cs:line 40
   at Microsoft.Azure.WebJobs.Script.WebHost.FileMonitoringService.InitializeSecondaryFileWatchers() in /src/azure-functions-host/src/WebJobs.Script.WebHost/FileMonitoringService.cs:line 192
   at Microsoft.Azure.WebJobs.Script.Utility.ExecuteAfterColdStartDelay(IEnvironment environment, Action targetAction, CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script/Utility.cs:line 728
   at Microsoft.Azure.WebJobs.Script.WebHost.FileMonitoringService.InitializeFileWatchers() in /src/azure-functions-host/src/WebJobs.Script.WebHost/FileMonitoringService.cs:line 173
   at Microsoft.Azure.WebJobs.Script.WebHost.FileMonitoringService.StartAsync(CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script.WebHost/FileMonitoringService.cs:line 94
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation, Int32 attemptCount, JobHostStartupMode startupMode) in /src/azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs:line 292

Investigative information

The deployment manifest configures the pod with the following security context:

        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 10001
          runAsGroup: 10001

The function app does not host any functions at all. I tried narrowing down the root cause and it lead me to try with various triggers until I had to remove all functions from the app. However, the error still occurs.

Repro steps

Here is the Dockerfile that builds the function when creating the image:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS installer-env

COPY ./FunctionApp /src/dotnet-function-app
COPY ./nuget.conf /src/dotnet-function-app/nuget.config
COPY ./packages /src/packages 
RUN cd /src/dotnet-function-app && \
    mkdir -p /home/site/wwwroot && \
    dotnet restore --runtime linux-x64 && \
    dotnet publish --configuration Release --runtime linux-x64 --self-contained true --output /home/site/wwwroot --no-restore

FROM mcr.microsoft.com/azure-functions/dotnet:3.0
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
    COMPlus_EnableDiagnostics=0

## Run as non root user

WORKDIR /home/site/wwwroot

RUN addgroup --group app --gid 10001 \
 && useradd --uid 10001 --gid 10001 "app" \
 && chown app:app /home/site/wwwroot

USER app:app

COPY --from=installer-env ["/home/site/wwwroot", "/home/site/wwwroot"]

Please, note that the COMPlus_EnableDiagnostics environment variable MUST be set to 0 (either in the Dockerfile or in the K8S deployment manifest) otherwise, the function fails to even start with the following, cryptic error message:

Failed to create CoreCLR, HRESULT: 0x80004005

Related information

Provide any related information

• Programming language used : C#
• Bindings used: None

[fabiocav]

@pragnagopa do you have public guidance on the images/requirements needed for standalone K8S hosting scenarios?

[pragnagopa]

@lpapudippu - can you help with the guidance needed?

[lpapudippu]

@jainharsh98 Can you please look into it?

[jainharsh98]

The issue is that during host startup, it attempts to create a directory here. The path for the directory is set here. By default it points to the /tmp folder. Since root file system is mounted as read-only, it is not possible to create the ‘Host’ directory in /tmp/Functions path leading to the exception.

This can be resolved by providing an ephemeral volume that can be written to. For example - mounting an emptyDir volume with mount path as /tmp allows the 'Host' directory creation and successful host startup.

  spec:
      containers:
      - name: netcoreapp
        image: jainharsh98/netcoreapp:1.0.1
        volumeMounts:
        - mountPath: /tmp
          name: tmpvol
        env:
        - name: ASPNETCORE_URLS
          value: http://+:8080
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 10001
          runAsGroup: 10001
      volumes:
      - emptyDir: {}
        name: tmpvol

Instead of using the /tmp folder for mounting, we can point FUNCTIONS_LOG_PATH variable to any suitable volume as well. For example -

  spec:
    containers:
    - name: netcoreapp
      image: jainharsh98/netcoreapp:1.0.1
      volumeMounts:
      - mountPath: /tmp1
        name: tmpvol
      env:
      - name: ASPNETCORE_URLS
        value: http://+:8080
      - name: FUNCTIONS_LOG_PATH
        value: /tmp1
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
    volumes:
    - emptyDir: {}
      name: tmpvol

The reason for choosing emptyDir is that it is safe across container crashes. It can be configured to consume resources on disk as well as memory.

Regarding COMPlus_EnableDiagnostics this seems to be documented as a netcore requirement.

Also, since the container is running with non-root permissions the ASPNETCORE_URLS environment variable should be over-written as by default kestrel server defaults to port 80 but binding any port < 1024 in linux requires root permission.

@fabiocav I see this directory is used for monitoring purpose. Can you please explain its exact use? Also are there any special restrictions or requirements necessary for this directory like should it persist even when the host restarts etc.

[springcomp]

That’s awesome. Thanks for getting to the bottom of this.

[lenndewolten]

@jainharsh98 Thanks for your response. This fixed indeed our first startup error. Unfortunately the host still breaks when trying to create the dir for the secretsSentinelFilePath in the BaseSecretsRepository.cs

System.IO.IOException: Read-only file system : '/azure-functions-host/Secrets/Sentinels'
         at System.IO.FileSystem.CreateDirectory(String fullPath)
         at System.IO.Directory.CreateDirectory(String path)
         at System.IO.Abstractions.DirectoryWrapper.CreateDirectory(String path)
         at Microsoft.Azure.WebJobs.Script.FileUtility.EnsureDirectoryExists(String path) in /src/azure-functions-host/src/WebJobs.Script/Extensions/FileUtility.cs:line 38

Applying the ENV setting FUNCTIONS_SECRETS_PATH to the volumeMount solve this issue for us.

[brettsam]

@lpapudippu, @jainharsh98 -- this seems like a documentation issue, would you agree? Is there some official place we can document these requirements if you want to use a read-only file system?

[jainharsh98]

@brettsam Yes, this does require documentation for the requirements and constraints when using a read-only file system with k8s for self-hosted function apps. We are currently working on validating these requirements and compiling a guidance document. Will update this thread accordingly.