haskell.nix Inconsistent hashes from source-repository-package

We are using amazonka via source-repository-package. Here's an example of the source-repository-package section:

source-repository-package
    type: git
    location: https://github.com/brendanhay/amazonka.git
    tag: ca99f52e595467f45183992449a8b14b31c349ae
    subdir:
        amazonka
        amazonka-sqs
        core

Several projects refer to the same tag while the sets of subdirs are different. Apparently this causes cache poisoning:

 error: hash mismatch importing path '/nix/store/gcji0zfhz0k7lgi5izr8zny3rrwp23ci-source-repository-package';
         specified: sha256:0v1i11rrxwfcxcr71a417lrj3mqkfnfpw38fcp61apn857ybzq0l
         got:       sha256:068fh6ig7a4fjva1srgqzzxkn6i1pxljjm5ic58rhfnsksw69x5z

Our workaround so far is to pick a different tag so that they don't conflict. I'm not familiar with the implementation detail but do you know where the problem is?

Jun 08 '22 02:06 fumieval

Can you elaborate a bit more on what you're doing? Are you talking about several projects using haskell.nix on... the same computer? Or what?

Jun 08 '22 08:06 michaelpj

Yes, several projects use haskell.nix in continuous integration, sharing /nix/store via S3. We noticed that this is something to do with amazonka; it's somewhat reproducible since the problem occurred even after deleting /nix/store/gcji0zfhz0k7lgi5izr8zny3rrwp23ci-source-repository-package.

Jun 08 '22 10:06 fumieval

Are you using sha256map or otherwise setting the expected sha for haskell.nix?

Jun 08 '22 11:06 michaelpj

Ah right, I'm using cabalProject so sha256map contains the hash for each source-repository-package.

Jun 08 '22 12:06 fumieval

Okay, so I would expect the different instances to have different sha256 values, right?

Jun 08 '22 12:06 michaelpj

Oh no, it should be the same since it's just for the git fetch. Not sure what's up...

Jun 08 '22 12:06 michaelpj

I wonder if the size of the repository (~1.4GB) is somehow related...

Jun 08 '22 13:06 fumieval

Check amazon for file and directory names that only differ in case. That can cause problems if some of the build machines have case insensitive file systems (the default on macOS).

Jun 09 '22 00:06 hamishmack

Just cloned amazonka on macOS to look into another issue and noticed git warned me:

the following paths have collided (e.g. case-sensitive paths
on a case-insensitive filesystem) and only one from the same
colliding group is in the working tree:
  'amazonka-iot/fixture/CreateCertificateFromCSR.yaml'
  'amazonka-iot/fixture/CreateCertificateFromCsr.yaml'
  'amazonka-iot/fixture/CreateCertificateFromCSRResponse.proto'
  'amazonka-iot/fixture/CreateCertificateFromCsrResponse.proto'

Jun 17 '22 07:06 hamishmack

Interesting... That could indeed be related

Jun 17 '22 13:06 fumieval

Checking my understanding: I'm guessing that this means that the output of fetchgit-ing amazonka is just going to have different hashes if you put nix on a case-insensitive filesystem? Ouch.

Jun 17 '22 14:06 michaelpj

We are using amazonka via source-repository-package. Here's an example of the source-repository-package section:
source-repository-package
    type: git
    location: https://github.com/brendanhay/amazonka.git
    tag: ca99f52e595467f45183992449a8b14b31c349ae
    subdir:
        amazonka
        amazonka-sqs
        core
Several projects refer to the same tag while the sets of subdirs are different. Apparently this causes cache poisoning:
 error: hash mismatch importing path '/nix/store/gcji0zfhz0k7lgi5izr8zny3rrwp23ci-source-repository-package';
         specified: sha256:0v1i11rrxwfcxcr71a417lrj3mqkfnfpw38fcp61apn857ybzq0l
         got:       sha256:068fh6ig7a4fjva1srgqzzxkn6i1pxljjm5ic58rhfnsksw69x5z
Our workaround so far is to pick a different tag so that they don't conflict. I'm not familiar with the implementation detail but do you know where the problem is?

Jun 18 '22 16:06 Vmc43hub

Well?

Jun 30 '22 06:06 Vmc43hub

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Oct 28 '22 11:10 stale[bot]

I'm not seeing this these days

Oct 31 '22 07:10 fumieval