llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard

Published 20 hours ago • verified publisher icon ggml-org

sha256 check sums to verify original and converted model data

Open gjmulder opened this issue 1 year ago • 5 comments

Not a developer, so my git-fu is a bit rusty. Hopefully this pull request covers everything?!?

  • Add shadow ./model.sha256 dir containing a dir for each model and a corresponding checklist.sha256 containing sha256 sums of the *.pth bin and *json files

  • Add script ./model.sha256/chk_sha256sums.sh to walk user-supplied ./models subdir and run sha256sum against above files to diff checklist.sha256 for each model

  • Update README.md with corresponding instructions

gjmulder avatar Mar 20 '23 20:03 gjmulder

Good idea, but I have one suggestions.

How about we put all hashes into a single file models/SHA256SUMS?

That's more standard way how to do it and you can then perform a single check using:

sha256sum -c models/SHA256SUMS

on Linux, or on macOS:

shasum -a 256 -c models/SHA256SUMS

That way you can drop the shellscript completely.

prusnak avatar Mar 20 '23 21:03 prusnak

@prusnak Thx! RTFM on sha256sum would have saved me an hour of unnecessary scripting. :facepalm:

I put the SHA256SUMS in the root dir as models is user-provided. I'm also not sure if there's a naming convention for the alpaca models, so I am implicitly proposing one.

llama.cpp$ sha256sum -c SHA256SUMS
models/13B/consolidated.00.pth: OK
models/13B/consolidated.01.pth: OK
models/30B/consolidated.00.pth: OK
models/30B/consolidated.01.pth: OK
models/30B/consolidated.02.pth: OK
models/30B/consolidated.03.pth: OK
models/65B/consolidated.00.pth: OK
models/65B/consolidated.01.pth: OK
models/65B/consolidated.02.pth: OK
models/65B/consolidated.03.pth: OK
models/65B/consolidated.04.pth: OK
models/65B/consolidated.05.pth: OK
models/65B/consolidated.06.pth: OK
models/65B/consolidated.07.pth: OK
models/7B/consolidated.00.pth: OK
models/13B/params.json: OK
models/30B/params.json: OK
models/65B/params.json: OK
models/7B/params.json: OK
models/alpaca-30B/params.json: OK
models/13B/ggml-model-f16.bin: OK
models/13B/ggml-model-f16.bin.1: OK
models/30B/ggml-model-f16.bin: OK
models/30B/ggml-model-f16.bin.1: OK
models/30B/ggml-model-f16.bin.2: OK
models/30B/ggml-model-f16.bin.3: OK
models/65B/ggml-model-f16.bin: OK
models/65B/ggml-model-f16.bin.1: OK
models/65B/ggml-model-f16.bin.2: OK
models/65B/ggml-model-f16.bin.3: OK
models/65B/ggml-model-f16.bin.4: OK
models/65B/ggml-model-f16.bin.5: OK
models/65B/ggml-model-f16.bin.6: OK
models/65B/ggml-model-f16.bin.7: OK
models/7B/ggml-model-f16.bin: OK
models/13B/ggml-model-q4_0.bin: OK
models/13B/ggml-model-q4_0.bin.1: OK
models/30B/ggml-model-q4_0.bin: OK
models/30B/ggml-model-q4_0.bin.1: OK
models/30B/ggml-model-q4_0.bin.2: OK
models/30B/ggml-model-q4_0.bin.3: OK
models/65B/ggml-model-q4_0.bin: OK
models/65B/ggml-model-q4_0.bin.1: OK
models/65B/ggml-model-q4_0.bin.2: OK
models/65B/ggml-model-q4_0.bin.3: OK
models/65B/ggml-model-q4_0.bin.4: OK
models/65B/ggml-model-q4_0.bin.5: OK
models/65B/ggml-model-q4_0.bin.6: OK
models/65B/ggml-model-q4_0.bin.7: OK
models/7B/ggml-model-q4_0.bin: OK
models/alpaca-13B/ggml-model-q4_0.bin: OK
models/alpaca-30B/ggml-model-q4_0.bin: OK
models/alpaca-7B/ggml-model-q4_0.bin: OK

gjmulder avatar Mar 21 '23 14:03 gjmulder

I put the SHA256SUMS in the root dir as models is user-provided

Great!

Can you please sort the file (according to filenames) using sort -k 2 SHA256SUMS > foo ; mv foo SHA256SUMS and update the PR?

prusnak avatar Mar 21 '23 14:03 prusnak

Good job, thanks a lot :)

I personally recommend renaming SHA256SUMS as models.sha256, with the following reasons:

  • Prefix with models, so it's clear that this file belongs to models.
  • Just like .sha1, it's clear that the .sha256 file contains sh256 checksum(s).
  • Prefer lower case if possible, because upper case is a bit hard to read.

mqy avatar Mar 21 '23 14:03 mqy

The filename SHA256SUMS is pretty standard and people are used to it. Examples:

  • https://releases.ubuntu.com/22.10/
  • https://bitcoincore.org/bin/bitcoin-core-24.0.1

Also later we can add more hashes to the file, not strictly related to the models, so naming it models.sha256 seems wrong.

prusnak avatar Mar 21 '23 15:03 prusnak

Can you add --ignore-missing to the sha256sum command line in the readme? Many people won't have all files, the output is less noisy that way. (This is for Linux, don't know about MacOS).

I had a check failure with models/7B/ggml-model-q4_0.bin, I think a recent commit may have lead to some floating point rounding differences? I have b85058443e89dabdf674d5018d979f0d682977f8413f05b5fd235d36d7a8ff82 for that file.

sw avatar Mar 21 '23 19:03 sw

Can you add --ignore-missing to the sha256sum command line in the readme? Many people won't have all files, the output is less noisy that way. (This is for Linux, don't know about MacOS).

I had a check failure with models/7B/ggml-model-q4_0.bin, I think a recent commit may have lead to some floating point rounding differences? I have b85058443e89dabdf674d5018d979f0d682977f8413f05b5fd235d36d7a8ff82 for that file.

@sw I just regenerated everything from the *.pth files and now our checksums agree:

llama.cpp$ git log | head -1
commit 353ec251a42491f5192c48561da4b444ef67f23c
llama.cpp$ grep b85058443e89dabdf674d5018d979f0d682977f8413f05b5fd235d36d7a8ff82 SHA256SUMS 
b85058443e89dabdf674d5018d979f0d682977f8413f05b5fd235d36d7a8ff82  models/7B/ggml-model-q4_0.bin

gjmulder avatar Mar 21 '23 20:03 gjmulder

(This is for Linux, don't know about MacOS).

Yes, it works for macOS too. I updated my suggestion here to contain the option: https://github.com/ggerganov/llama.cpp/pull/338#discussion_r1143489500

prusnak avatar Mar 21 '23 21:03 prusnak

I went ahead and implemented the suggestions from above and rebased/squashed on top of the current master.

prusnak avatar Mar 21 '23 22:03 prusnak

Thanks @gjmulder for computing the hashes. Merged!

prusnak avatar Mar 21 '23 22:03 prusnak

Not all of these checksums seem to be correct. Are they calculated with the "v2" new model format after the tokenizer change? PR: https://github.com/ggerganov/llama.cpp/pull/252 Issue: https://github.com/ggerganov/llama.cpp/issues/324

For example, "models/alpaca-7B/ggml-model-q4_0.bin"

v1: 1f582babc2bd56bb63b33141898748657d369fd110c4358b2bc280907882bf13 v2: 8d5562ec1d8a7cfdcf8985a9ddf353339d942c7cf52855a92c9ff59f03b541bc

The SHA256SUMS file has the old v1 hash. Maybe using a naming scheme like "ggml2-model-q4_0.bin" would be good to differentiate between the versions and avoid confusion.

anzz1 avatar Mar 21 '23 22:03 anzz1

I can confirm ggml-model-q4* in 7B and 13B mismatch that in SHA256SUMS. Local checksums:

  • b85058443e89dabdf674d5018d979f0d682977f8413f05b5fd235d36d7a8ff82 models/7B/ggml-model-q4_0.bin
  • d3ab1548a2d19d989c1e7ea1130ab8c6300c75941a434a2d333ef564f989131b models/13B/ggml-model-q4_0.bin
  • 38b705ce6c5baba4bb6056f11189a4ad21b70591258422450e30495b6ccd8521 models/13B/ggml-model-q4_0.bin.1

Other files in 7B and 13B are correct. I regenerated mismatched q4 files with latest program (make clean; make).

mqy avatar Mar 21 '23 23:03 mqy

Please open new pull requests if something is wrong.

Maybe using a naming scheme like "ggml2-model-q4_0.bin" would be good to differentiate between the versions and avoid confusion.

We should keep only the latest hashes in the SHA256SUM file, generated by the latest version of the tools in the repo. Introducing any versioning scheme can lead to even more confusion. And if you need to check the older hashes you can still check the earlier versions of the SHA256SUM file. Ideally, the same commit which changes the file format will also regenerate hashes.

prusnak avatar Mar 22 '23 08:03 prusnak

Not all of these checksums seem to be correct. Are they calculated with the "v2" new model format after the tokenizer change? PR: #252 Issue: #324

For example, "models/alpaca-7B/ggml-model-q4_0.bin"

v1: 1f582babc2bd56bb63b33141898748657d369fd110c4358b2bc280907882bf13 v2: 8d5562ec1d8a7cfdcf8985a9ddf353339d942c7cf52855a92c9ff59f03b541bc

The SHA256SUMS file has the old v1 hash. Maybe using a naming scheme like "ggml2-model-q4_0.bin" would be good to differentiate between the versions and avoid confusion.

Yes, that was why I was delaying merging this pull request. See the model magic and versioning discussion in #352:

llama.cpp/models$ cat chk_versions.sh 
#!/bin/sh

for B in *B/ggml-model*bin*; do
	xxd $B | head -1 | awk -v model=$B '{printf("Model: %30s, magic: 0x%8s, version: 0x%4s\n", model, $3$2, $4)}'
done
llama.cpp/models$ ./chk_versions.sh | sort -nk 2
Model:  alpaca-7B/ggml-model-q4_0.bin, magic: 0x67676c6d, version: 0x007d
Model: alpaca-13B/ggml-model-q4_0.bin, magic: 0x67676c6d, version: 0x007d
Model: alpaca-30B/ggml-model-q4_0.bin, magic: 0x67676c6d, version: 0x007d
Model:          7B/ggml-model-f16.bin, magic: 0x6767666d, version: 0x0100
Model:         7B/ggml-model-q4_0.bin, magic: 0x6767666d, version: 0x0100
Model:         13B/ggml-model-f16.bin, magic: 0x6767666d, version: 0x0100
Model:        13B/ggml-model-q4_0.bin, magic: 0x6767666d, version: 0x0100
Model:       13B/ggml-model-f16.bin.1, magic: 0x6767666d, version: 0x0100
Model:      13B/ggml-model-q4_0.bin.1, magic: 0x6767666d, version: 0x0100
Model:         30B/ggml-model-f16.bin, magic: 0x6767666d, version: 0x0100
Model:        30B/ggml-model-q4_0.bin, magic: 0x6767666d, version: 0x0100
Model:       30B/ggml-model-f16.bin.1, magic: 0x6767666d, version: 0x0100
Model:       30B/ggml-model-f16.bin.2, magic: 0x6767666d, version: 0x0100
Model:       30B/ggml-model-f16.bin.3, magic: 0x6767666d, version: 0x0100
Model:      30B/ggml-model-q4_0.bin.1, magic: 0x6767666d, version: 0x0100
Model:      30B/ggml-model-q4_0.bin.2, magic: 0x6767666d, version: 0x0100
Model:      30B/ggml-model-q4_0.bin.3, magic: 0x6767666d, version: 0x0100
Model:         65B/ggml-model-f16.bin, magic: 0x6767666d, version: 0x0100
Model:        65B/ggml-model-q4_0.bin, magic: 0x6767666d, version: 0x0100
Model:       65B/ggml-model-f16.bin.1, magic: 0x6767666d, version: 0x0100
Model:       65B/ggml-model-f16.bin.2, magic: 0x6767666d, version: 0x0100
Model:       65B/ggml-model-f16.bin.3, magic: 0x6767666d, version: 0x0100
Model:       65B/ggml-model-f16.bin.4, magic: 0x6767666d, version: 0x0100
Model:       65B/ggml-model-f16.bin.5, magic: 0x6767666d, version: 0x0100
Model:       65B/ggml-model-f16.bin.6, magic: 0x6767666d, version: 0x0100
Model:       65B/ggml-model-f16.bin.7, magic: 0x6767666d, version: 0x0100
Model:      65B/ggml-model-q4_0.bin.1, magic: 0x6767666d, version: 0x0100
Model:      65B/ggml-model-q4_0.bin.2, magic: 0x6767666d, version: 0x0100
Model:      65B/ggml-model-q4_0.bin.3, magic: 0x6767666d, version: 0x0100
Model:      65B/ggml-model-q4_0.bin.4, magic: 0x6767666d, version: 0x0100
Model:      65B/ggml-model-q4_0.bin.5, magic: 0x6767666d, version: 0x0100
Model:      65B/ggml-model-q4_0.bin.6, magic: 0x6767666d, version: 0x0100
Model:      65B/ggml-model-q4_0.bin.7, magic: 0x6767666d, version: 0x0100

gjmulder avatar Mar 22 '23 10:03 gjmulder