dotnet lambda package is not reproducible

[rittneje]

Describe the bug

Because dotnet lambda package does not zero out the mtimes of the files in the zip, the zip file itself constantly changes on every build. This means that even rerunning a build of the same exact commit will lead to a cdk diff.

Expected Behavior

dotnet lambda package should produce identical results given identical inputs.

Current Behavior

The original mtimes are preserved, so there is always a diff.

Reproduction Steps

1. Run dotnet lambda package.
2. Use cdk deploy to deploy a stack with that lambda function.
3. Run dotnet lambda package again.
4. Run cdk diff.

Possible Solution

Set all mtimes to zero (or provide a flag for this).

Additional Information/Context

No response

Targeted .NET platform

.NET Core 3.1

CLI extension version

amazon.lambda.tools 4.0.0 dotnet-lambda

Environment details (OS name and version, etc.)

Debian

[ashishdhingra]

@rittneje Good afternoon. I'm not sure if this is an issue. The dotnet lambda package command internally calls dotnet publish to produce the output. It would (and should) not modify any attributes of the files generates as part of dotnet publish command. It would be interesting to see which files you see have modified timestamps as a result of running the command again. Is it the output of the dotnet publish command?

[rittneje]

The problem is not that it modified the file attributes. In fact, it's the opposite. It is required for dotnet lambda package to set all the mtimes to some static value before generating the zip file, or else the zip file contents will always change, in turn leading to spurious cdk diffs.

[ashishdhingra]

It is required for dotnet lambda package to set all the mtimes to some static value before generating the zip file, or else the zip file contents will always change, in turn leading to spurious cdk diffs.

@rittneje So are you saying it is not doing it right now and should do it (i.e. set all the mtimes to some static value) as a feature request? I would also advise to set proper issue title for easy review.

[rittneje]

Regardless of whether you want to consider it a bug or a feature request, the problem is that dotnet lambda package is currently incompatible with cdk. And it is incompatible because it does not allow for byte-for-byte reproducible builds (hence the issue title).

[Dreamescaper]

Probably a duplicate of https://github.com/aws/aws-extensions-for-dotnet-cli/issues/195.

[lowkasen]

Facing the same issue and haven't found a fix

[ma7tz]

Hi, I had the same problem. But in the end I found a solution:

dotnet lambda package lambda.zip \
  --configuration Release \
  /p:Deterministic=true \
  /p:DebugType=None \
  /p:IncludeSourceRevisionInInformationalVersion=false

In addition, the timestamps had to be eliminated. Fortunately, there is a ready solution for this:

strip-nondeterminism -v ./lambda.zip

If you are on a debian system, you can install it easily:

apt-get install -y strip-nondeterminism

After that, cdk did not detect any changes in successive pipeline runs if no source code was changed.

I checked the hash values ​​in the pipeline to always identify the cause of changes:

shasum ./lambda.zip
strip-nondeterminism -v ./lambda.zip
shasum ./lambda.zip
mkdir ./files && unzip ./lambda.zip -d ./files/ && find ./files/${i%/} -type f -exec shasum {} \; && rm -rf ./files

• https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/code-generation#deterministic
• https://learn.microsoft.com/en-us/dotnet/core/compatibility/sdk/8.0/source-link
• https://salsa.debian.org/reproducible-builds/strip-nondeterminism