convert.py fails on the dolly model with KeyError: ('torch', 'ByteStorage')

[devkral]

Prerequisites

I'm trying to use the truly opensource dolly 2.0 model with lama.cpp. For converting the pytorch bin to ggml I use the convert.py script in the repository.

Please answer the following questions for yourself before submitting an issue.

• [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [x] I carefully followed the README.md.
• [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [x] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

./convert.py either fails with a meaningful error or converts the model to ggml

Current Behavior

The process fails with the cryptic message:

KeyError: ('torch', 'ByteStorage')

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  6
  On-line CPU(s) list:   0-5
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 5 4500U with Radeon Graphics
    CPU family:          23
    Model:               96
    Thread(s) per core:  1
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            1
    Frequency boost:     enabled
    CPU(s) scaling MHz:  64%
    CPU max MHz:         2375,0000
    CPU min MHz:         1400,0000
    BogoMIPS:            4743,75
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxe
                         xt fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq
                          monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_lega
                         cy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwa
                         itx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx s
                         map clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf x
                         saveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausef
                         ilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca
Virtualization features: 
  Virtualization:        AMD-V
Caches (sum of all):     
  L1d:                   192 KiB (6 instances)
  L1i:                   192 KiB (6 instances)
  L2:                    3 MiB (6 instances)
  L3:                    8 MiB (2 instances)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-5
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT disabled
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

• Operating System, e.g. for Linux:

$ uname -a

Linux mobarch2 6.2.11-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 13 Apr 2023 16:59:24 +0000 x86_64 GNU/Linux

• SDK version, e.g. for Linux:

$ python3 --version
Python 3.10.10

$ make --version
GNU Make 4.4.1
Built for x86_64-pc-linux-gnu

$ g++ --version
g++ (GCC) 12.2.1 20230201

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. installed requirements in a virtual environment
2. switched to the environment
3. tried to either install torch in the venv or to use the system version of it
4. python ./convert.py --dump ./models/dolly-v2-12b_pytorch.bin

Failure Logs

Loading model file models/dolly-v2-12b_pytorch.bin
Traceback (most recent call last):
  File "/home/alex/git/llama.cpp/./convert.py", line 1149, in <module>
    main()
  File "/home/alex/git/llama.cpp/./convert.py", line 1129, in main
    model_plus = load_some_model(args.model)
  File "/home/alex/git/llama.cpp/./convert.py", line 1055, in load_some_model
    models_plus.append(lazy_load_file(path))
  File "/home/alex/git/llama.cpp/./convert.py", line 849, in lazy_load_file
    return lazy_load_torch_file(fp, path)
  File "/home/alex/git/llama.cpp/./convert.py", line 721, in lazy_load_torch_file
    model = unpickler.load()
  File "/home/alex/git/llama.cpp/./convert.py", line 710, in find_class
    return self.CLASSES[(module, name)]
KeyError: ('torch', 'ByteStorage')

[devkral]

I found the problem:

dolly uses uint8 datatypes and they are not mapped.

I think it is a byte Tensor which maps to uint8 which causes the problems

[badvision]

I was able to follow how I32 is handled and repeat those same changes for I8 to map to ByteStorage, etc. However the next stumbling block is that it doesn't want to load the tokenizer model. I tried renaming tokenizer.json to tokenizer.model but that just results in a parsing error.

Diff of my changes:

brobert@mac llama.cpp % diff convert.py convert_orig.py
45d44
< DT_I8 = UnquantizedDataType('I8')
74d72
<     DT_I8: np.dtype(np.int8)
707d704
<         ('torch', 'ByteStorage'): LazyStorageKind(DT_I8)
733d729
<     'I8': DT_I8

[devkral]

yeah that is an other issue: the tokens for dolly are in json format and convert handles some other format.

[tdipadova3rd]

@devkral Did you end up figuring this out?

[ggaabe]

I could definitely use this as well if anyone's figured it out

[devkral]

yes, see closed PR: https://github.com/ggerganov/llama.cpp/pull/1308

[Green-Sky]

this is the wrong repository :) see here https://github.com/ggerganov/ggml/tree/master/examples/dolly-v2

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.