[ECS Fargate] [Bug]: Set Environment variables at Machine level in Windows Container when ECS Task Spins up

[vijayaram-b]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about the Issue We have a Windows Service application which currently runs on the on-prem environments, we are in the process of migrating the on-prem environments to the AWS Cloud and using the AWS ECS Fargate services and availing a few of the services in the existing Application.

We have modified our application to write the log into the Cloud Watch using the AWS SDK rather than writing to the local file (current approach) when we deployed the container we got the below error.

 **Unable to get IAM security credentials from EC2 Instance Metadata Service.**

In order to find the root cause of this issue, we have written a POC application and run it is a console application. When we deployed this we did not get any error. So we have updated the same logic to the POC windows service application and still got the same error. So we end up printing all the environment variables in both applications to make sure we are getting all the values correctly.

Find the Screenshot below for reference

As per the image, we could see the environment variables we set as part of the Task Definition (LOG_GROUP_NAME & LOG_STREAM_NAME) along with the AWS Environment variables are not populating in the Windows Service Application.

Based on this We believe that when ECS spins up a Task all the environment variables are set only at the default user level, not the machine level which kind of breaks our code as we have a windows service which will run as a LocalSystem account and does not have access to the user level Environment variable

Are you currently working around this issue? For the time being, we have developed a sample C# program which executes the PowerShell script Get-ChildItem Env: -Include 'AWS*', 'ECS*' -Recurse and set these to Machine level environment variables to make it work during the Start-Up

Environment Details

• Launch Type - Fargate (For our actual solution, for POC purposes we even tried this in the ECS EC2 and had the same issue)
• Role - We have provided admin access for both Task and Execution Role
• Operating System Family - Windows Server 2019 Full
• Platform Version - LATEST (1.0.0)
• Memory 8GB, CPU - 4
• Container Base Image - mcr.microsoft.com/windows/servercore:ltsc2019-amd64
• .Net Framework 4.8 which we have installed on top of the Container Base Image

AWS Case ID We even discussed the same issue with the AWS Team and the details can be found in Case ID: 10253345431

Attachments

• Docker File referenced from https://github.com/microsoft/dotnet-framework-docker/blob/a853e05e409b3009b059fa3945837765c9bc43b8/src/runtime/4.8/windowsservercore-ltsc2022/Dockerfile

[vijayaram-b]

Shared the POC Application and along with the logs requested by AWS Team in the same case id: 10253345431

[jterry75]

@vijayaram-b - Unfortunately, Windows containers do not support setting environment variables in a container at machine scope. The environment variables will only be applied to the processes started by the exec service. (Meaning init, or any children of init). To set an environment variable at machine level it needs to be in the registry which can be done with a Dockerfile build process but it cannot then contain dynamic data like you want here.

As a workaround, you could have an init script started in your container that simply takes its environment and applies at machine scope and then starts your Windows service. I know its a workaround, but it would unblock support for your scenario I believe.

[vijayaram-b]

Thank you @jterry75 , can this be added as a feature when spinning up the windows container?

[jterry75]

Not really. On Windows, there is no shared Environment namespace across Containers like there is for Networking. What this means is that in order for ECS/Fargate to do this the agent would have to run its own process in the customer workload container which is not owned by ECS. For networking we use an ECS owned "pause" container to setup the environment so we explicitly do not run code in the customer container but again this would not work for Environment since setting a machine level env in this "pause" container would not affect the Workload containers environment in the task.

[vijayaram-b]

So basically, we should avoid going to AWS ECS Fargate Windows if we have a .NET Windows Service application as we will not be able to get either the AWS Environment variables or the ones we have set, is my understanding correct?

[jterry75]

@vijayaram-b - That is not what I said.

Windows containers support machine level environment variables via exactly one mechanism, which is the Windows registry set statically via the build process. This is true on any platform hosting Windows containers because it is a feature of the Microsoft API provided, not of the hosting runtime such as ECS. In a Dockerfile if you you add an ENV it will be placed in the Windows registry for that container image. When the Windows container is started it will then have the Machine level environment variable. Microsoft does not provide a mechanism to place dynamic environment variables into the container at runtime at Machine level. It will only place those dynamic environment variables at the scope of processes started by the container exec service within the container namespace. These processes are your init process of the container, and any sidecar processes that you exec in the lifetime of the container.

In order for ECS (or any runtime) to set these without a Microsoft provided API it would require that some process inside your container image allows the runtime to call into it to set these from within the container namespace. This violates our promise of customer trust as we do not run any code not owned by the customer within the container.

As mentioned above, you can use Windows Services and fix this issue rather easily if you change your init process to be a simple PowerShell script that copies the environment variables set dynamically from ECS into the Machine level scope and then calls Start-Service for each of your services within the container. This gives you control of how you would like to handle pushing the environment variables, which ones your service needs, while also maintaining our customer trust.

I would be happy to formally ask Microsoft for this feature on your behalf if you would like and you can +1 the ask if you would like.

-Justin

[davidcarson]

@jterry75 you're a life-saver!

Solved my problem with a .NET Framework 4.8 / IIS / ECS Fargate Windows container not being able to talk to DynamoDb by putting something like this in my entrypoint..

$ecsContainerMetadataUri = [Environment]::GetEnvironmentVariable('ECS_CONTAINER_METADATA_URI', 'Process')
$awsContainerCredentialsRelativeUri = [Environment]::GetEnvironmentVariable('AWS_CONTAINER_CREDENTIALS_RELATIVE_URI', 'Process')

[Environment]::SetEnvironmentVariable('ECS_CONTAINER_METADATA_URI', $ecsContainerMetadataUri, 'Machine')
[Environment]::SetEnvironmentVariable('AWS_CONTAINER_CREDENTIALS_RELATIVE_URI', $awsContainerCredentialsRelativeUri, 'Machine')

iisreset /restart

[to0b]

@davidcarson Can you please share your entrypoint script and your docker entrypoint definition for reference?

My Windows .NET/IIS Fargate ECS task always stops after executing the entrypoint ps1 script where I'm doing something similar:

foreach($key in [System.Environment]::GetEnvironmentVariables('Process').Keys) {
  if ([System.Environment]::GetEnvironmentVariable($key, 'Machine') -eq $null ) {
    $val = [System.Environment]::GetEnvironmentVariable($key, 'Process');
    [System.Environment]::SetEnvironmentVariable($key, $val, 'Machine')
  }
}

[davidcarson]

Hi @to0b ,

@davidcarson Can you please share your entrypoint script and your docker entrypoint definition for reference?

My entrypoint script does a couple of things only relevant to our app, then the environment variable copying, and then:

while ($true) { Start-Sleep -Seconds 3600 }

Can't remember where I picked this up from, but it should keep the ECS task running.

And the Dockerfile has:

ENTRYPOINT ["powershell", ".\\InitializeContainer"]

(where InitializeContainer.ps1 is my entrypoint script)