Patch for TL1 kernel generation setup on Arm64 (Mac M1 Pro).

[patches-ml]

Hey all!

I am attempting to run and benchmark TL1 inference on my Mac M1 Pro device. After following the guide for the build process successfully, I ran python setup_env.py -q TL1 (after installing a pre-converted GGUF model from HuggingFace) to specify dynamic generation of TL1 kernels during inference. However, this setup process is littered with bugs. To give a few examples:

• In the hf-to-gguf conversion script, a registration decorator is applied to the general BitNet model class, but there is a mismatch in capitalisation. Currently it is: @Model.register("BitnetForCausalLM") which I changed to @Model.register("BitNetForCausalLM"). This was a simple fix.
• Further, the converter by default assumes the model uses a SentencePiece tokeniser, despite a GPT-2 style tokeniser being specified in the logs. There is no clear selection of code paths separating the appropriate tokeniser, the BitNet model class immediately uses the SentencePiece tokeniser initialisation method. I changed this temporarily to call the GPT-2 tokeniser included in the same script but this is a temporary fix.
• Furthermore, kernel configuration fails. Specifically, BM=-1. This means kernel_config.ini does not include a shape which conforms to the built-in M and K constants. I can add these in manually but this feels like a haphazard fix.

So, in summary:

• Am I doing something obviously wrong which is generating all these errors?
• I will attempt to patch kernel configuration, but why do the shapes not automatically conform?
• Is this a known issue for this device and/or architecture?
• Assuming this is a legitimate collection of bugs, can I submit a patch, if I am able to solve the kernel configuration problem? Otherwise, would it be possible for the team working on this framework to provide a more robust and universal solution?

This is the full log (some of the file paths have been obscured):

Traceback (most recent call last): File "~/BitNet/utils/convert-hf-to-gguf-bitnet.py", line 1175, in main() File "~BitNet/utils/convert-hf-to-gguf-bitnet.py", line 1167, in main model_instance.write() File "~/BitNet/utils/convert-hf-to-gguf-bitnet.py", line 200, in write self.write_tensors() File "~/BitNet/utils/convert-hf-to-gguf-bitnet.py", line 1055, in write_tensors data, i2_scale = transform_to_tl1(data) ^^^^^^^^^^^^^^^^^^^^^^ File "~/BitNet/utils/convert-hf-to-gguf-bitnet.py", line 667, in transform_to_tl1 res = preprocess_weights_tl1(x) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "~/BitNet/utils/convert-hf-to-gguf-bitnet.py", line 509, in preprocess_weights_tl1 raise NotImplementedError NotImplementedError

Thank you!

[maksi65432]

it likely moment when clang gone wrong from clang 17 it sometimes have this bug

[patches-ml]

Thanks for the response @maksi65432. Could you tell me if there are a list of known working builds of BitNet.cpp , recommended clang version, etc.