haskell.nix icon indicating copy to clipboard operation
haskell.nix copied to clipboard

Published 20 hours ago • verified publisher icon input-output-hk

Inconsistent hashes from source-repository-package

Open fumieval opened this issue 2 years ago • 13 comments

We are using amazonka via source-repository-package. Here's an example of the source-repository-package section:

source-repository-package
    type: git
    location: https://github.com/brendanhay/amazonka.git
    tag: ca99f52e595467f45183992449a8b14b31c349ae
    subdir:
        amazonka
        amazonka-sqs
        core

Several projects refer to the same tag while the sets of subdirs are different. Apparently this causes cache poisoning:

 error: hash mismatch importing path '/nix/store/gcji0zfhz0k7lgi5izr8zny3rrwp23ci-source-repository-package';
         specified: sha256:0v1i11rrxwfcxcr71a417lrj3mqkfnfpw38fcp61apn857ybzq0l
         got:       sha256:068fh6ig7a4fjva1srgqzzxkn6i1pxljjm5ic58rhfnsksw69x5z

Our workaround so far is to pick a different tag so that they don't conflict. I'm not familiar with the implementation detail but do you know where the problem is?

fumieval avatar Jun 08 '22 02:06 fumieval

Can you elaborate a bit more on what you're doing? Are you talking about several projects using haskell.nix on... the same computer? Or what?

michaelpj avatar Jun 08 '22 08:06 michaelpj

Yes, several projects use haskell.nix in continuous integration, sharing /nix/store via S3. We noticed that this is something to do with amazonka; it's somewhat reproducible since the problem occurred even after deleting /nix/store/gcji0zfhz0k7lgi5izr8zny3rrwp23ci-source-repository-package.

fumieval avatar Jun 08 '22 10:06 fumieval

Are you using sha256map or otherwise setting the expected sha for haskell.nix?

michaelpj avatar Jun 08 '22 11:06 michaelpj

Ah right, I'm using cabalProject so sha256map contains the hash for each source-repository-package.

fumieval avatar Jun 08 '22 12:06 fumieval

Okay, so I would expect the different instances to have different sha256 values, right?

michaelpj avatar Jun 08 '22 12:06 michaelpj

Oh no, it should be the same since it's just for the git fetch. Not sure what's up...

michaelpj avatar Jun 08 '22 12:06 michaelpj

I wonder if the size of the repository (~1.4GB) is somehow related...

fumieval avatar Jun 08 '22 13:06 fumieval

Check amazon for file and directory names that only differ in case. That can cause problems if some of the build machines have case insensitive file systems (the default on macOS).

hamishmack avatar Jun 09 '22 00:06 hamishmack

Just cloned amazonka on macOS to look into another issue and noticed git warned me:

the following paths have collided (e.g. case-sensitive paths
on a case-insensitive filesystem) and only one from the same
colliding group is in the working tree:
  'amazonka-iot/fixture/CreateCertificateFromCSR.yaml'
  'amazonka-iot/fixture/CreateCertificateFromCsr.yaml'
  'amazonka-iot/fixture/CreateCertificateFromCSRResponse.proto'
  'amazonka-iot/fixture/CreateCertificateFromCsrResponse.proto'

hamishmack avatar Jun 17 '22 07:06 hamishmack

Interesting... That could indeed be related

fumieval avatar Jun 17 '22 13:06 fumieval

Checking my understanding: I'm guessing that this means that the output of fetchgit-ing amazonka is just going to have different hashes if you put nix on a case-insensitive filesystem? Ouch.

michaelpj avatar Jun 17 '22 14:06 michaelpj

We are using amazonka via source-repository-package. Here's an example of the source-repository-package section:

source-repository-package
    type: git
    location: https://github.com/brendanhay/amazonka.git
    tag: ca99f52e595467f45183992449a8b14b31c349ae
    subdir:
        amazonka
        amazonka-sqs
        core

Several projects refer to the same tag while the sets of subdirs are different. Apparently this causes cache poisoning:

 error: hash mismatch importing path '/nix/store/gcji0zfhz0k7lgi5izr8zny3rrwp23ci-source-repository-package';
         specified: sha256:0v1i11rrxwfcxcr71a417lrj3mqkfnfpw38fcp61apn857ybzq0l
         got:       sha256:068fh6ig7a4fjva1srgqzzxkn6i1pxljjm5ic58rhfnsksw69x5z

Our workaround so far is to pick a different tag so that they don't conflict. I'm not familiar with the implementation detail but do you know where the problem is?

Vmc43hub avatar Jun 18 '22 16:06 Vmc43hub

Well?

Vmc43hub avatar Jun 30 '22 06:06 Vmc43hub

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Oct 28 '22 11:10 stale[bot]

I'm not seeing this these days

fumieval avatar Oct 31 '22 07:10 fumieval