added hashes for weights and tokenizer

[dylancvdean]

[igorbrigadir]

Are these derived from the files in the torrent or from the official download?

[dylancvdean]

These are from the official download they sent me

[vukosim]

The universe is healing

[dylancvdean]

Should use SHA-256, Blake2B or other modern digest instead of MD5 that is known to be broken.

Good point, computing that now

[ztgasdf]

Not sure why you would use BLAKE2b when BLAKE3 is a lot faster.

[Tronic]

Not sure why you would use BLAKE2b when BLAKE3 is a lot faster.

A fair point, especially in this case since the data is huge and Blake3 is much, much faster. One might use the older digests I mentioned due to them having better existing software support, but yes, I would prefer Blake3 as well.

[dylancvdean]

I'll do b3sum next, seems useful to have the fastest and the most readily available

[lowqualitybot]

SHA256 of the files in the torrent linked in PR #73 882e6431d0b08a8bc66261a0d3607da21cbaeafa96a24e7e59777632dbdac225 ./65B/consolidated.04.pth d27f5b0677d7ff129ceacd73fd461c4d06910ad7787cf217b249948c3f3bc638 ./65B/consolidated.07.pth 999ed1659b469ccc2a941714c0a9656fa571d17c9f7c8c7589817ca90edef51b ./65B/params.json a287c0dfe49081626567c7fe87f74cce5831f58e459b427b5e05567641f47b78 ./65B/consolidated.05.pth 135c563f6b3938114458183afb01adc9a63bef3d8ff7cccc3977e5d3664ecafe ./65B/consolidated.00.pth e7babf7c5606f165a3756f527cb0fedc4f83e67ef1290391e52fb1cce5f26770 ./65B/consolidated.02.pth 72b4eba67a1a3b18cb67a85b70f8f1640caae9b40033ea943fb166bd80a7b36b ./65B/consolidated.06.pth 9a600b37b19d38c7e43809485f70d17d1dc12206c07efa83bc72bb498a568bde ./65B/consolidated.01.pth cfc27c8b98c3a9c02db876353adf3c9593608f014701cd248fd0dd84b1d60c7d ./65B/checklist.chk 73176ffb426b40482f2aa67ae1217ef79fbbd1fff5482bae5060cdc5a24ab70e ./65B/consolidated.03.pth 4ab77bec4d4405ccb66a97b282574c89a94417e3c32e5f68f37e2876fc21322f ./13B/params.json 745bf4e29a4dd6f411e72976d92b452da1b49168a4f41c951cfcc8051823cf08 ./13B/consolidated.00.pth d5ccbcc465c71c0de439a5aeffebe8344c68a519bce70bc7f9f92654ee567085 ./13B/consolidated.01.pth 183eb00cea5c880fd88c296af1038f4c15dc26aa2ccb7c6cf2c35b9bb00dce45 ./13B/checklist.chk 7e89e242ddc0dd6f060b43ca219ce8b3e8f08959a72cb3c0855df8bb04d46265 ./7B/params.json 700df0d3013b703a806d2ae7f1bfb8e59814e3d06ae78be0c66368a50059f33d ./7B/consolidated.00.pth 7935c843a25ae265d60bf4543b90bfd91c4911b728412b5c1d5cff42a3cd5645 ./7B/checklist.chk 2c07118ea98d69dbe7810d88520e30288fa994751b337f8fca02b171955f44cb ./30B/params.json e23294a58552d8cdec5b7e8abb87993b97ea6eced4178ff2697c02472539d067 ./30B/consolidated.00.pth 24a87f01028cbd3a12de551dcedb712346c0b5cbdeff1454e0ddf2df9b675378 ./30B/consolidated.02.pth 4e077b7136c7ae2302e954860cf64930458d3076fcde9443f4d0e939e95903ff ./30B/consolidated.01.pth e1d6c07bff756afa6d1f40665cb5e028aa993c4d761cdecffb0ff4cb51513fa6 ./30B/checklist.chk 1adfcef71420886119544949767f6a56cb6339b4d5fcde755d80fe68b49de93b ./30B/consolidated.03.pth c5e05c673c6c32276ea1b27e413ba460e864d42cf487d50246402baf5b7cb9b0 ./llama.sh 021e52e6d404bf3b84008be179dada01e4627b4801bfbc73a88a030f229c6b34 ./tokenizer_checklist.chk 9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347 ./tokenizer.model

[dylancvdean]

SHA256 of the files in the torrent linked in PR #73 882e6431d0b08a8bc66261a0d3607da21cbaeafa96a24e7e59777632dbdac225 ./65B/consolidated.04.pth d27f5b0677d7ff129ceacd73fd461c4d06910ad7787cf217b249948c3f3bc638 ./65B/consolidated.07.pth 999ed1659b469ccc2a941714c0a9656fa571d17c9f7c8c7589817ca90edef51b ./65B/params.json a287c0dfe49081626567c7fe87f74cce5831f58e459b427b5e05567641f47b78 ./65B/consolidated.05.pth 135c563f6b3938114458183afb01adc9a63bef3d8ff7cccc3977e5d3664ecafe ./65B/consolidated.00.pth e7babf7c5606f165a3756f527cb0fedc4f83e67ef1290391e52fb1cce5f26770 ./65B/consolidated.02.pth 72b4eba67a1a3b18cb67a85b70f8f1640caae9b40033ea943fb166bd80a7b36b ./65B/consolidated.06.pth 9a600b37b19d38c7e43809485f70d17d1dc12206c07efa83bc72bb498a568bde ./65B/consolidated.01.pth cfc27c8b98c3a9c02db876353adf3c9593608f014701cd248fd0dd84b1d60c7d ./65B/checklist.chk 73176ffb426b40482f2aa67ae1217ef79fbbd1fff5482bae5060cdc5a24ab70e ./65B/consolidated.03.pth 4ab77bec4d4405ccb66a97b282574c89a94417e3c32e5f68f37e2876fc21322f ./13B/params.json 745bf4e29a4dd6f411e72976d92b452da1b49168a4f41c951cfcc8051823cf08 ./13B/consolidated.00.pth d5ccbcc465c71c0de439a5aeffebe8344c68a519bce70bc7f9f92654ee567085 ./13B/consolidated.01.pth 183eb00cea5c880fd88c296af1038f4c15dc26aa2ccb7c6cf2c35b9bb00dce45 ./13B/checklist.chk 7e89e242ddc0dd6f060b43ca219ce8b3e8f08959a72cb3c0855df8bb04d46265 ./7B/params.json 700df0d3013b703a806d2ae7f1bfb8e59814e3d06ae78be0c66368a50059f33d ./7B/consolidated.00.pth 7935c843a25ae265d60bf4543b90bfd91c4911b728412b5c1d5cff42a3cd5645 ./7B/checklist.chk 2c07118ea98d69dbe7810d88520e30288fa994751b337f8fca02b171955f44cb ./30B/params.json e23294a58552d8cdec5b7e8abb87993b97ea6eced4178ff2697c02472539d067 ./30B/consolidated.00.pth 24a87f01028cbd3a12de551dcedb712346c0b5cbdeff1454e0ddf2df9b675378 ./30B/consolidated.02.pth 4e077b7136c7ae2302e954860cf64930458d3076fcde9443f4d0e939e95903ff ./30B/consolidated.01.pth e1d6c07bff756afa6d1f40665cb5e028aa993c4d761cdecffb0ff4cb51513fa6 ./30B/checklist.chk 1adfcef71420886119544949767f6a56cb6339b4d5fcde755d80fe68b49de93b ./30B/consolidated.03.pth c5e05c673c6c32276ea1b27e413ba460e864d42cf487d50246402baf5b7cb9b0 ./llama.sh 021e52e6d404bf3b84008be179dada01e4627b4801bfbc73a88a030f229c6b34 ./tokenizer_checklist.chk 9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347 ./tokenizer.model

That's a match

[USBhost]

Based.

[dbuos]

Do you have some script to execute the match? In order to avoid manual checks?

[mapleroyal]

Do you have some script to execute the match? In order to avoid manual checks?

If you're in linux, the following command will print all the b3sum's to your terminal.

find /path/to/directory -type f | while read file; do b3sum "$file"; done

Here's ChatGPT's explanation of what it does:

This is a command executed in a Unix-like shell environment, which does the following:

1. find /path/to/directory -type f searches for all the files under the directory /path/to/directory and its subdirectories recursively, excluding any directories found in the search. The -type f option limits the search to only files.

2. The pipe character | takes the output of the find command and passes it as input to the next command.

3. The while read file; do loop reads each line of output from the find command and assigns it to the variable $file.

4. b3sum "$file" calculates the BLAKE3 hash value of the file represented by $file.

5. done signals the end of the loop.

Therefore, the overall command finds all the files under the specified directory and subdirectories, calculates the BLAKE3 hash value for each file, and prints the hash values to the console.

[Tronic]

It's a bit cumbersome that each file has a separate b3sum file. If all were listed in a single file LLaMa.b3sum at the root directory, you could simply run

b3sum --check LLaMa.b3sum

Or possibly there could be a separate file for each folder, since not everyone wants to download those largest models. You can do --check *.b3sum too, so it isn't that bad anyway. No need to use find+while and pipes.