Colab Pro - invalid load key

[BenHamm]

When running the demo code, with 13B Vicunia, I encounter an error when trying to load the model checkpoint. Seee the following stdout:

2023-04-25 03:08:17.957341: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Initializing Chat Loading VIT Loading VIT Done Loading Q-Former Loading Q-Former Done Loading LLAMA

bin /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/lib/python3.9/dist-packages/cv2/../../lib64')} warn(msg) /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: /usr/local/lib/python3.9/dist-packages/cv2/../../lib64:/usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')} warn(msg) /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.28.0.1'), PosixPath('http'), PosixPath('8013')} warn(msg) /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-v100-hm-38eqcqbw7bonk --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')} warn(msg) /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')} warn(msg) /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')} warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward. Either way, this might cause trouble in the future: If you get CUDA error: invalid device function errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env. warn(msg) CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0 CUDA SETUP: Highest compute capability among GPUs detected: 7.0 CUDA SETUP: Detected CUDA version 118 /usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU! warn(msg) CUDA SETUP: Loading binary /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so... Loading checkpoint shards: 100% 3/3 [02:08<00:00, 42.82s/it] Loading LLAMA Done Load 4 training prompts Prompt Example ###Human: <Img><ImageHere></Img> Describe this image in detail. ###Assistant: Load BLIP2-LLM Checkpoint: pretrained_minigpt4.pth ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /content/MiniGPT-4/demo.py:60 in │ │ │ │ 57 model_config = cfg.model_cfg │ │ 58 model_config.device_8bit = args.gpu_id │ │ 59 model_cls = registry.get_model_class(model_config.arch) │ │ ❱ 60 model = model_cls.from_config(model_config).to('cuda:{}'.format(args.g │ │ 61 │ │ 62 vis_processor_cfg = cfg.datasets_cfg.cc_sbu_align.vis_processor.train │ │ 63 vis_processor = registry.get_processor_class(vis_processor_cfg.name).f │ │ │ │ /content/MiniGPT-4/minigpt4/models/mini_gpt4.py:265 in from_config │ │ │ │ 262 │ │ ckpt_path = cfg.get("ckpt", "") # load weights of MiniGPT-4 │ │ 263 │ │ if ckpt_path: │ │ 264 │ │ │ print("Load BLIP2-LLM Checkpoint: {}".format(ckpt_path)) │ │ ❱ 265 │ │ │ ckpt = torch.load(ckpt_path, map_location="cpu") │ │ 266 │ │ │ msg = model.load_state_dict(ckpt['model'], strict=False) │ │ 267 │ │ │ │ 268 │ │ return model │ │ │ │ /usr/local/lib/python3.9/dist-packages/torch/serialization.py:815 in load │ │ │ │ 812 │ │ │ │ return _legacy_load(opened_file, map_location, _weigh │ │ 813 │ │ │ except RuntimeError as e: │ │ 814 │ │ │ │ raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) │ │ ❱ 815 │ │ return _legacy_load(opened_file, map_location, pickle_module, │ │ 816 │ │ 817 │ │ 818 # Register pickling support for layout instances such as │ │ │ │ /usr/local/lib/python3.9/dist-packages/torch/serialization.py:1033 in │ │ _legacy_load │ │ │ │ 1030 │ │ │ f"Received object of type "{type(f)}". Please update to │ │ 1031 │ │ │ "functionality.") │ │ 1032 │ │ │ ❱ 1033 │ magic_number = pickle_module.load(f, **pickle_load_args) │ │ 1034 │ if magic_number != MAGIC_NUMBER: │ │ 1035 │ │ raise RuntimeError("Invalid magic number; corrupt file?") │ │ 1036 │ protocol_version = pickle_module.load(f, **pickle_load_args) │ ╰──────────────────────────────────────────────────────────────────────────────╯ UnpicklingError: invalid load key, '<'.

[nelsonjchen]

The wget instructions wget the HTML, not the raw download. Probably worth a PR to fix that wget.

[nelsonjchen]

@WangRongsheng 's wget instructions in the Colab, to be more specific.