Provide Documentation on Securing ASP.NET Core Containers

[RehanSaeed]

I've discovered that you can run an ASP.NET Core image with a read-only file system but this requires you to turn off debugging and profiling support because otherwise you get an error. Full sample here.

docker run --rm --read-only -it -p 8000:80 -e COMPlus_EnableDiagnostics=0 my-asp-app

There are also a myriad of settings we can use in Kubernetes. Here is a sample Pod yaml:

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: gcr.io/google-samples/node-hello:1.0
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        add: ["NET_ADMIN", "SYS_TIME"]

• allowPrivilegeEscalation - Stops escalation of privlieges to super user.
• readOnlyRootFilesystem - Enables the read-only file system I talk about above.
• runAsUser - Run as a different user.
• fsGroup - Run as a different group.
• capabilities - Limit the linux capabilities available to the app.

As a linux noob, I'd really like more information and guidance on runAsUser, fsGroup and capabilities in particular. It would be ideal if a basic set of capabilities could be provided to get a hello world app running but also some description of what needs to be added to get additional features.

[MichaelSimons]

@richlander - There are some good suggestions here. Is this something better covered by the docs or a sample here?

[RehanSaeed]

I'd just like to add that running a read-only file system does not currently work if you are buffering requests in ASP.NET Core. If you are transferring JSON in ASP.NET Core, then JSON.NET apparently requires buffering but the new JSON API's in .NET Core 3 will apparently solve this issue. More details in https://github.com/RehanSaeed/ReadOnlyDockerTest/issues/1 and https://github.com/aspnet/AspNetCore/issues/3704.

[RehanSaeed]

Also adding PID Limiting to this list which is a new feature in Kubernetes 1.14 but is currently beta. That feature will probably go live in 1.15.

[MichaelSimons]

@glennc - Is this something you are interested in documenting from the ASP.NET perspective if something doesn't already exist?

[MichaelSimons]

Ping @glennc

[MichaelSimons]

~~@blowdart will be taking on this per conversation started by @richlander. Feel free to move the issue to an ASP.NET or other repo if there is a more appropriate place to track this work.~~

<Edit> The topic of providing guidance still being discussed. No decisions have been made yet.

[blowdart]

Please don't move this to the aspnet repo, as the vast majority of this is not an aspnet question and I will simply move it back.

To speak to the one asp.net part of this;

"you can run an ASP.NET Core image with a read-only file system"

@RehanSaeed you are frankly lucky in what you tested. ASP.NET Core will start to buffer requests to disk when under memory pressure because not doing so is a huge DoS vector. ASP.NET does not support running on read only file systems. Just because it gets "fixed" in JSON doesn't mean it will get fixed in the request pipeline. It's currently not a goal.

[MichaelSimons]

@blowdart - sure - I have no intentions to move it myself. I am just trying to get @RehanSaeed question answered and a decision if we are going to provide additional guidance.

[hbiarge]

Are there any decision taken on this? Any advise over those security settings?

Thanks!

[baoduy]

My app using Service Bus Client and after applied the Security Context I got this issue on AKS. Please the solution to fix the issue if anyone have. Thank you so much

[41m[1m[37mcrit[39m[22m[49m: Microsoft.AspNetCore.Server.Kestrel[0]
      Unable to start Kestrel.
      System.Net.Sockets.SocketException (13): Permission denied
         at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
         at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
         at System.Net.Sockets.Socket.Bind(EndPoint localEP)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.<Bind>g__BindSocket|13_0(<>c__DisplayClass13_0& )
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass29_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IEnumerable`1 listenOptions, AddressBindContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
[40m[32minfo[39m[22m[49m: Balance.Infras.Events.BalanceSubscriptionClient[0]
      Handling TransferStatusChanged
[41m[30mfail[39m[22m[49m: Balance.Infras.Events.BalanceSubscriptionClient[0]
      Service bus client on balance-transfer-sub failed to handle message call back Cannot access a disposed object.
      Object name: 'IServiceProvider'. ; action UserCallback on endpoint sg-dev-backbonebus-trans.servicebus.windows.net and entity transfer-v1-tp/Subscriptions/balance-transfer-sub
      System.ObjectDisposedException: Cannot access a disposed object.
      Object name: 'IServiceProvider'.
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.CreateScope(IServiceProvider provider)
         at Balance.Infras.Events.BalanceSubscriptionClient.<RegisterSubscriptionClientMessageHandler>b__17_0(Message message, CancellationToken token) in /app/Balance/Balance.Infras/Events/BalanceSubscriptionClient.cs:line 114
         at Microsoft.Azure.ServiceBus.MessageReceivePump.MessageDispatchTask(Message message)
Unhandled exception. System.Net.Sockets.SocketException (13): Permission denied
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.<Bind>g__BindSocket|13_0(<>c__DisplayClass13_0& )
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass29_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IEnumerable`1 listenOptions, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Balance.Api.Program.Main(String[] args) in /app/Balance/Balance.Api/Program.cs:line 57
   at Balance.Api.Program.<Main>(String[] args)

[RehanSaeed]

There is a nice writeup here. Any chance this could be turned into proper docs?

https://techcommunity.microsoft.com/t5/azure-developer-community-blog/hardening-an-asp-net-container-running-on-kubernetes/ba-p/2542224

[mthalman]

@davidfowl - Does there exist any documentation that might help this request of having guidance for securely configuring ASP.NET Core in container scenarios?

[davidfowl]

cc @richlander

[danmoseley]

@RehanSaeed you are frankly lucky in what you tested. ASP.NET Core will start to buffer requests to disk when under memory pressure because not doing so is a huge DoS vector. ASP.NET does not support running on read only file systems. Just because it gets "fixed" in JSON doesn't mean it will get fixed in the request pipeline. It's currently not a goal.

Is the question assuming that /tmp (or $ASPNETCORE_TEMP) is not writeable? or only that the deployment directory is not writeable?

[benjaminlhai]

This needs more attention - diagnostics / profiling should not be mutually exclusive with read only root filesystem. At least allow the usage of a custom directory/mount a tmpfs or similar volume for collection and set that folder via environment vars. Unless this is an option that is documented somewhere? I couldn't find anything valuable (other than this issue)

[esbenbjerre]

@richlander can you confirm whether or not it is considered safe to run a .NET 7/8 app on a read-only file system assuming ASPNETCORE_TEMP points to a writeable folder?

[davidfowl]

It should be yes.

[richlander]

I have another blog post planned on prod deployments. I can include this aspect.