llama icon indicating copy to clipboard operation
llama copied to clipboard

Published 20 hours ago • verified publisher icon meta-llama

Approved, but unable to download weights

Open delip opened this issue 1 year ago • 56 comments

When I run the download.sh I see this. image

And I don't see any *.pth files in the download directory.

image

Any suggestions?

delip avatar Mar 02 '23 04:03 delip

I had the same issue downloading locally (Mac Os Monterey) but seems to be working fine in AWS (ubuntu)

dafraile avatar Mar 02 '23 04:03 dafraile

I edited the code to handle that exception in dictionary defining on mac.

# Copyright (c) Meta Platforms, Inc. and affiliates.
# This software may be used and distributed according to the terms of the GNU General Public License version 3.

PRESIGNED_URL=""             # replace with presigned url from email
MODEL_SIZE="7"  # edit this list with the model sizes you wish to download
TARGET_FOLDER="download"             # where all files should end up

declare -A N_SHARD_DICT

N_SHARD_DICT["7"]="0"
N_SHARD_DICT["13"]="1"
N_SHARD_DICT["30"]="3"
N_SHARD_DICT["65"]="7"

echo "Downloading tokenizer"
wget ${PRESIGNED_URL/'*'/"tokenizer.model"} -O ${TARGET_FOLDER}"/tokenizer.model"
wget ${PRESIGNED_URL/'*'/"tokenizer_checklist.chk"} -O ${TARGET_FOLDER}"/tokenizer_checklist.chk"

(cd ${TARGET_FOLDER} && md5 tokenizer_checklist.chk)

for i in ${MODEL_SIZE//,/ }
do
    echo "Downloading ${i}B"
    mkdir -p ${TARGET_FOLDER}"/${i}B"
    for s in $(seq -f "0%g" 0 ${N_SHARD_DICT[$i]})
    do
        echo "Downloading shard ${s}"
        wget ${PRESIGNED_URL/'*'/"${i}B/consolidated.${s}.pth"} -O ${TARGET_FOLDER}"/${i}B/consolidated.${s}.pth"
    done
    wget ${PRESIGNED_URL/'*'/"${i}B/params.json"} -O ${TARGET_FOLDER}"/${i}B/params.json"
    wget ${PRESIGNED_URL/'*'/"${i}B/checklist.chk"} -O ${TARGET_FOLDER}"/${i}B/checklist.chk"
    echo "Checking checksums"
    (cd ${TARGET_FOLDER}"/${i}B" && md5 checklist.chk)
done

ramithuh avatar Mar 02 '23 04:03 ramithuh

I end up getting either an Error 400: Bad Request, or a 403 when I replace the link. Just to confirm, it is a cloudfront link right?

Screenshot 2023-03-01 at 10 49 47 PM

sumukshashidhar avatar Mar 02 '23 04:03 sumukshashidhar

Same problem on macOS. besides, I got the extra error:

download.sh: line 9: declare: -A: invalid option

Hannibal046 avatar Mar 02 '23 04:03 Hannibal046

@sumukshashidhar it's a cloudfront link. Curious why yours show a urldefence.com link.. maybe the network is blocking ?

ramithuh avatar Mar 02 '23 04:03 ramithuh

I end up getting either an Error 400: Bad Request, or a 403 when I replace the link. Just to confirm, it is a cloudfront link right?

Screenshot 2023-03-01 at 10 49 47 PM

Fixed it. Institution replaces all links with a urldefense link.

I was able to resolve it by opening the link in a web browser, and copying the new link after I see the XML.

sumukshashidhar avatar Mar 02 '23 04:03 sumukshashidhar

@Hannibal046 I had got the same error message, solved it by installing wget and md5sum (coreutils).

arqamrp avatar Mar 02 '23 05:03 arqamrp

@arqamrp got it! Thanks

Hannibal046 avatar Mar 02 '23 05:03 Hannibal046

For the error

download.sh: line 9: declare: -A: invalid option

I used

brew install bash

to install brew v5 and then use the bash installed by brew to download (in my case /opt/homebrew/Cellar/bash/5.2.15/bin/bash download.sh).

sapphire008 avatar Mar 02 '23 05:03 sapphire008

Screenshot 2023-03-02 at 1 19 36 PM Anyone else got this error?

yanzhangnlp avatar Mar 02 '23 05:03 yanzhangnlp

I am getting a 403 forbidden error:

HTTP request sent, awaiting response... 403 Forbidden
2023-03-02 05:39:58 ERROR 403: Forbidden.

Any suggestion would be appreciated? : /

basujindal avatar Mar 02 '23 05:03 basujindal

I am getting a 403 forbidden error:

HTTP request sent, awaiting response... 403 Forbidden
2023-03-02 05:39:58 ERROR 403: Forbidden.

Any suggestion would be appreciated? : /

you cannot use curl or wget directly.

tekspirit avatar Mar 02 '23 05:03 tekspirit

I am getting a 403 forbidden error:

HTTP request sent, awaiting response... 403 Forbidden
2023-03-02 05:39:58 ERROR 403: Forbidden.

Any suggestion would be appreciated? : /

you cannot use curl or wget directly.

Hi, thanks for the reply, I used the download.sh file with the received link (even removed the url defence) but I am still getting the error on Ubuntu 22

basujindal avatar Mar 02 '23 05:03 basujindal

I am getting a 403 forbidden error:

HTTP request sent, awaiting response... 403 Forbidden
2023-03-02 05:39:58 ERROR 403: Forbidden.

Any suggestion would be appreciated? : /

you cannot use curl or wget directly.

Hi, thanks for the reply, I used the download.sh file with the received link (even removed the url defence) but I am still getting the error on Ubuntu 22

did you use bash? bash download.sh

tekspirit avatar Mar 02 '23 05:03 tekspirit

I am getting a 403 forbidden error:

HTTP request sent, awaiting response... 403 Forbidden
2023-03-02 05:39:58 ERROR 403: Forbidden.

Any suggestion would be appreciated? : /

you cannot use curl or wget directly.

Hi, thanks for the reply, I used the download.sh file with the received link (even removed the url defence) but I am still getting the error on Ubuntu 22

did you use bash? bash download.sh

Yes, I did

basujindal avatar Mar 02 '23 05:03 basujindal

I'm getting the checklist.chk and the params.json but the consolidated.xx.pth files are not coming down or do not exist. Any ideas about this?

tekspirit avatar Mar 02 '23 06:03 tekspirit

I tried to download it through mobaxterm on windows 10, and got the same error. I fixed the error by installing coreutils.

apt install coreutils

G-U-N avatar Mar 02 '23 06:03 G-U-N

Using aria2 instead of wget

zhongtao93 avatar Mar 02 '23 07:03 zhongtao93

I edited the code to handle that exception in dictionary defining on mac.

# Copyright (c) Meta Platforms, Inc. and affiliates.
# This software may be used and distributed according to the terms of the GNU General Public License version 3.

PRESIGNED_URL=""             # replace with presigned url from email
MODEL_SIZE="7"  # edit this list with the model sizes you wish to download
TARGET_FOLDER="download"             # where all files should end up

declare -A N_SHARD_DICT

N_SHARD_DICT["7"]="0"
N_SHARD_DICT["13"]="1"
N_SHARD_DICT["30"]="3"
N_SHARD_DICT["65"]="7"

echo "Downloading tokenizer"
wget ${PRESIGNED_URL/'*'/"tokenizer.model"} -O ${TARGET_FOLDER}"/tokenizer.model"
wget ${PRESIGNED_URL/'*'/"tokenizer_checklist.chk"} -O ${TARGET_FOLDER}"/tokenizer_checklist.chk"

(cd ${TARGET_FOLDER} && md5 tokenizer_checklist.chk)

for i in ${MODEL_SIZE//,/ }
do
    echo "Downloading ${i}B"
    mkdir -p ${TARGET_FOLDER}"/${i}B"
    for s in $(seq -f "0%g" 0 ${N_SHARD_DICT[$i]})
    do
        echo "Downloading shard ${s}"
        wget ${PRESIGNED_URL/'*'/"${i}B/consolidated.${s}.pth"} -O ${TARGET_FOLDER}"/${i}B/consolidated.${s}.pth"
    done
    wget ${PRESIGNED_URL/'*'/"${i}B/params.json"} -O ${TARGET_FOLDER}"/${i}B/params.json"
    wget ${PRESIGNED_URL/'*'/"${i}B/checklist.chk"} -O ${TARGET_FOLDER}"/${i}B/checklist.chk"
    echo "Checking checksums"
    (cd ${TARGET_FOLDER}"/${i}B" && md5 checklist.chk)
done

I had the same problem and this works! Thanks @ramithuh

harrywang avatar Mar 02 '23 15:03 harrywang

Here is a version of download.sh that works with /bin/zsh (zsh 5.8.1 (x86_64-apple-darwin22.0)

# Copyright (c) Meta Platforms, Inc. and affiliates.
# This software may be used and distributed according to the terms of the GNU General Public License version 3.

PRESIGNED_URL=""             # replace with presigned url from email
MODEL_SIZE="X7B,X13B,X30B,X65B"  # edit this list with the model sizes you wish to download
TARGET_FOLDER="."             # where all files should end up

declare -a N_SHARD_DICT

N_SHARD_DICT["X7B"]="0"
N_SHARD_DICT["X13B"]="1"
N_SHARD_DICT["X30B"]="3"
N_SHARD_DICT["X65B"]="7"

echo "Downloading tokenizer"
wget ${PRESIGNED_URL/'*'/"tokenizer.model"} -O ${TARGET_FOLDER}"/tokenizer.model"
wget ${PRESIGNED_URL/'*'/"tokenizer_checklist.chk"} -O ${TARGET_FOLDER}"/tokenizer_checklist.chk"

(cd ${TARGET_FOLDER} && md5 -r tokenizer_checklist.chk)

for i in ${MODEL_SIZE//,/ }
do
    m="${i:1}" # remove preceding X
    echo "Downloading ${m}"
    mkdir -p ${TARGET_FOLDER}"/${m}"
    for s in $(seq -f "0%g" 0 ${N_SHARD_DICT[$i]})
    do
        echo ${PRESIGNED_URL/'*'/"${m}/consolidated.${s}.pth"} -O ${TARGET_FOLDER}"/${m}/consolidated.${s}.pth"
        wget ${PRESIGNED_URL/'*'/"${m}/consolidated.${s}.pth"} -O ${TARGET_FOLDER}"/${m}/consolidated.${s}.pth"
    done
    wget ${PRESIGNED_URL/'*'/"${m}/params.json"} -O ${TARGET_FOLDER}"/${m}/params.json"
    wget ${PRESIGNED_URL/'*'/"${m}/checklist.chk"} -O ${TARGET_FOLDER}"/${m}/checklist.chk"
    echo "Checking checksums"
    (cd ${TARGET_FOLDER}"/${m}" && md5 -r checklist.chk)
done

drewgillson avatar Mar 02 '23 16:03 drewgillson

Connecting to dobf1k6cxlizq.cloudfront.net|13.33.45.217|:443... connected. ERROR: cannot verify dobf1k6cxlizq.cloudfront.net's certificate, issued by `/C=US/O=Amazon/CN=Amazon RSA 2048 M01': Hello anyone meet this error?

kli017 avatar Mar 03 '23 03:03 kli017

How to run in mac M1? Did anyone try it?

definev avatar Mar 03 '23 04:03 definev

i got approved but unable to download the 7B weight , now please tell what steps should be followed in order to access the 7B weight i have installed all the tools required for it , thanking you

riskyhomo avatar Mar 03 '23 12:03 riskyhomo

i got approved but unable to download the 7B weight , now please tell what steps should be followed in order to access the 7B weight i have installed all the tools required for it , thanking you

If you on mac, there is a script here in the issues you can use that works.

tekspirit avatar Mar 03 '23 20:03 tekspirit

I end up getting either an Error 400: Bad Request, or a 403 when I replace the link. Just to confirm, it is a cloudfront link right?

Screenshot 2023-03-01 at 10 49 47 PM

https://github.com/facebookresearch/llama/issues/57#issuecomment-1455086931

This seems to be a problem of the network region. Set the proxy to JP or US to solve this problem

trekrollercoaster avatar Mar 06 '23 03:03 trekrollercoaster

Still getting ERROR 403: Forbidden.. How to fix it? Tried to change region to US - no luck.

Vitaly-Protasov avatar Mar 06 '23 09:03 Vitaly-Protasov

Just wrote a python script to replicate what would be done in bash

from os import path, makedirs, getcwd, chdir
from argparse import ArgumentParser
import logging
import subprocess

logging.basicConfig(level=logging.DEBUG)

p = ArgumentParser("Download LLaMA")
p.add_argument('--check-only', '-c', action='store_true', help="Only doing File")
p.add_argument('--dryrun', '-d', action='store_true', help="Dryrun, only print but no execute")
p.add_argument('--target', '-o', default='downloads', type=str, help="Target output diretory")
p.add_argument('--models', '-m', default=["7B","13B","30B","65B"], nargs='+', help="Models you want to download")
args = p.parse_args()

N_SHARD_DICT = {}
N_SHARD_DICT["7B"] = 0
N_SHARD_DICT["13B"] = 1
N_SHARD_DICT["30B"] = 3
N_SHARD_DICT["65B"] = 7

assert [m in ["7B","13B","30B","65B"] for m in args.models]
TARGET_FOLDER=lambda x: path.join(args.target, x)
PRESIGNED_URL=lambda x: f"https://<ur-prefix-of-presigned-url>.cloudfront.net/{x}?Policy=<ur-rest-of-presign-url>"

makedirs(TARGET_FOLDER(""), exist_ok=True)

models = ['tokenizer.model', "tokenizer_checklist.chk"]
for s in args.models:
    makedirs(TARGET_FOLDER(s), exist_ok=True)
    models.extend([f"{s}/consolidated.0{n}.pth" for n in range(N_SHARD_DICT[s]+1)])
    models.extend([f"{s}/{f}" for f in ["params.json", "checklist.chk"]])

for i, n in enumerate(models):
    logging.info(f"[{i}/{len(models)}]Downloding {n}...")
    wget_str = f"wget -c {PRESIGNED_URL(n)} -O {TARGET_FOLDER(n)}"
    if args.dryrun:
        logging.debug(f"-- {wget_str}")
    else:
        if not args.check_only:
            subprocess.run(wget_str.split(' '))


for i, s in enumerate(args.models):
    cmd = "md5sum -c checklist.chk"
    logging.info(f"[{i}/{len(args.models)}] Checking Model {s}...")
    if args.dryrun:
        logging.debug(f"-- {cmd}")
    else:
        subprocess.run(cmd.split(' '), cwd=TARGET_FOLDER(s))

mpskex avatar Mar 08 '23 05:03 mpskex

Just wrote a python script to replicate what would be done in bash

from os import path, makedirs, getcwd, chdir
from argparse import ArgumentParser
import logging
import subprocess

logging.basicConfig(level=logging.DEBUG)

p = ArgumentParser("Download LLaMA")
p.add_argument('--check-only', '-c', action='store_true', help="Only doing File")
p.add_argument('--dryrun', '-d', action='store_true', help="Dryrun, only print but no execute")
p.add_argument('--target', '-o', default='downloads', type=str, help="Target output diretory")
p.add_argument('--models', '-m', default=["7B","13B","30B","65B"], nargs='+', help="Models you want to download")
args = p.parse_args()

N_SHARD_DICT = {}
N_SHARD_DICT["7B"] = 0
N_SHARD_DICT["13B"] = 1
N_SHARD_DICT["30B"] = 3
N_SHARD_DICT["65B"] = 7

assert [m in ["7B","13B","30B","65B"] for m in args.models]
TARGET_FOLDER=lambda x: path.join(args.target, x)
PRESIGNED_URL=lambda x: f"https://dobf1k6cxlizq.cloudfront.net/{x}?Policy=<ur-rest-of-presign-url>"

makedirs(TARGET_FOLDER(""), exist_ok=True)

models = ['tokenizer.model', "tokenizer_checklist.chk"]
for s in args.models:
    makedirs(TARGET_FOLDER(s), exist_ok=True)
    models.extend([f"{s}/consolidated.0{n}.pth" for n in range(N_SHARD_DICT[s]+1)])
    models.extend([f"{s}/{f}" for f in ["params.json", "checklist.chk"]])

for i, n in enumerate(models):
    logging.info(f"[{i}/{len(models)}]Downloding {n}...")
    wget_str = f"wget -c {PRESIGNED_URL(n)} -O {TARGET_FOLDER(n)}"
    if args.dryrun:
        logging.debug(f"-- {wget_str}")
    else:
        if not args.check_only:
            subprocess.run(wget_str.split(' '))


for i, s in enumerate(args.models):
    cmd = "md5sum -c checklist.chk"
    logging.info(f"[{i}/{len(args.models)}] Checking Model {s}...")
    if args.dryrun:
        logging.debug(f"-- {cmd}")
    else:
        subprocess.run(cmd.split(' '), cwd=TARGET_FOLDER(s))

I was also getting the ERROR 403: Forbidden error when connecting to after using @ramithu 's above code. Reinstalling brew, alternative curl, wget and aria2 didn't work either. Tried company wifi, home wifi and personal hotspot, no success, still same error. However the above quoted code from @mpskex was successful. Just create a new download.py file, copy pasta, change lines 11 and 23 to your respective default TARGET_FOLDER and PRESIGNED_URL and it should work when you python download.py in terminal. Thank you @mpskex

However for the 7B and 13B models, the consolidated.00.pth file don't download with error:

Resolving dobf1k6cxlizq.cloudfront.net (dobf1k6cxlizq.cloudfront.net)... failed: nodename nor servname provided, or not known. wget: unable to resolve host address ‘dobf1k6cxlizq.cloudfront.net’

vincentwi avatar Mar 08 '23 17:03 vincentwi

Just wrote a python script to replicate what would be done in bash

from os import path, makedirs, getcwd, chdir
from argparse import ArgumentParser
import logging
import subprocess

logging.basicConfig(level=logging.DEBUG)

p = ArgumentParser("Download LLaMA")
p.add_argument('--check-only', '-c', action='store_true', help="Only doing File")
p.add_argument('--dryrun', '-d', action='store_true', help="Dryrun, only print but no execute")
p.add_argument('--target', '-o', default='downloads', type=str, help="Target output diretory")
p.add_argument('--models', '-m', default=["7B","13B","30B","65B"], nargs='+', help="Models you want to download")
args = p.parse_args()

N_SHARD_DICT = {}
N_SHARD_DICT["7B"] = 0
N_SHARD_DICT["13B"] = 1
N_SHARD_DICT["30B"] = 3
N_SHARD_DICT["65B"] = 7

assert [m in ["7B","13B","30B","65B"] for m in args.models]
TARGET_FOLDER=lambda x: path.join(args.target, x)
PRESIGNED_URL=lambda x: f"https://dobf1k6cxlizq.cloudfront.net/{x}?Policy=<ur-rest-of-presign-url>"

makedirs(TARGET_FOLDER(""), exist_ok=True)

models = ['tokenizer.model', "tokenizer_checklist.chk"]
for s in args.models:
    makedirs(TARGET_FOLDER(s), exist_ok=True)
    models.extend([f"{s}/consolidated.0{n}.pth" for n in range(N_SHARD_DICT[s]+1)])
    models.extend([f"{s}/{f}" for f in ["params.json", "checklist.chk"]])

for i, n in enumerate(models):
    logging.info(f"[{i}/{len(models)}]Downloding {n}...")
    wget_str = f"wget -c {PRESIGNED_URL(n)} -O {TARGET_FOLDER(n)}"
    if args.dryrun:
        logging.debug(f"-- {wget_str}")
    else:
        if not args.check_only:
            subprocess.run(wget_str.split(' '))


for i, s in enumerate(args.models):
    cmd = "md5sum -c checklist.chk"
    logging.info(f"[{i}/{len(args.models)}] Checking Model {s}...")
    if args.dryrun:
        logging.debug(f"-- {cmd}")
    else:
        subprocess.run(cmd.split(' '), cwd=TARGET_FOLDER(s))

I was also getting the ERROR 403: Forbidden error when connecting to after using @ramithu 's above code. Reinstalling brew, alternative curl, wget and aria2 didn't work either. Tried company wifi, home wifi and personal hotspot, no success, still same error. However the above quoted code from @mpskex was successful. Just create a new download.py file, copy pasta, change lines 11 and 23 to your respective default TARGET_FOLDER and PRESIGNED_URL and it should work when you python download.py in terminal. Thank you @mpskex

However for the 7B and 13B models, the consolidated.00.pth file don't download with error:

Resolving dobf1k6cxlizq.cloudfront.net (dobf1k6cxlizq.cloudfront.net)... failed: nodename nor servname provided, or not known. wget: unable to resolve host address ‘dobf1k6cxlizq.cloudfront.net’

I would suggest you replace the prefix of that url dobf1k6cxlizq.cloudfront.net into <prefix-in-your-url>.cloudfront.net. That may fix your problem.

mpskex avatar Mar 09 '23 04:03 mpskex

Just wrote a python script to replicate what would be done in bash

from os import path, makedirs, getcwd, chdir
from argparse import ArgumentParser
import logging
import subprocess

logging.basicConfig(level=logging.DEBUG)

p = ArgumentParser("Download LLaMA")
p.add_argument('--check-only', '-c', action='store_true', help="Only doing File")
p.add_argument('--dryrun', '-d', action='store_true', help="Dryrun, only print but no execute")
p.add_argument('--target', '-o', default='downloads', type=str, help="Target output diretory")
p.add_argument('--models', '-m', default=["7B","13B","30B","65B"], nargs='+', help="Models you want to download")
args = p.parse_args()

N_SHARD_DICT = {}
N_SHARD_DICT["7B"] = 0
N_SHARD_DICT["13B"] = 1
N_SHARD_DICT["30B"] = 3
N_SHARD_DICT["65B"] = 7

assert [m in ["7B","13B","30B","65B"] for m in args.models]
TARGET_FOLDER=lambda x: path.join(args.target, x)
PRESIGNED_URL=lambda x: f"https://<ur-prefix-of-presigned-url>.cloudfront.net/{x}?Policy=<ur-rest-of-presign-url>"

makedirs(TARGET_FOLDER(""), exist_ok=True)

models = ['tokenizer.model', "tokenizer_checklist.chk"]
for s in args.models:
    makedirs(TARGET_FOLDER(s), exist_ok=True)
    models.extend([f"{s}/consolidated.0{n}.pth" for n in range(N_SHARD_DICT[s]+1)])
    models.extend([f"{s}/{f}" for f in ["params.json", "checklist.chk"]])

for i, n in enumerate(models):
    logging.info(f"[{i}/{len(models)}]Downloding {n}...")
    wget_str = f"wget -c {PRESIGNED_URL(n)} -O {TARGET_FOLDER(n)}"
    if args.dryrun:
        logging.debug(f"-- {wget_str}")
    else:
        if not args.check_only:
            subprocess.run(wget_str.split(' '))


for i, s in enumerate(args.models):
    cmd = "md5sum -c checklist.chk"
    logging.info(f"[{i}/{len(args.models)}] Checking Model {s}...")
    if args.dryrun:
        logging.debug(f"-- {cmd}")
    else:
        subprocess.run(cmd.split(' '), cwd=TARGET_FOLDER(s))

I was also getting the 403 Forbidden error when running the bash script. This python script worked perfectly for me on Mac OS Ventura 13.0.1 M1 Pro processor. Thanks!

sidsvash26 avatar Mar 09 '23 11:03 sidsvash26