I help FlagAI build a HuggingFace version Aquila-7B/Aquilachat-7B

[qhduan]

Model description

I think you must be too busy recently, so the power of the community is needed at this time. I am not sure whether my implementation is completely correct, but I believe these tasks will help you

https://huggingface.co/qhduan/aquila-7b

https://huggingface.co/qhduan/aquilachat-7b

Open source status

• [X] The model implementation is available
• [X] The model weights are available

Provide useful links for the implementation

No response

[tonysy]

Great job! Can we use this implementation for batch inference ?

[qhduan]

Great job! Can we use this implementation for batch inference ?

I think you could do everything as long as you don't violate the license.

I didn't modify the code based on transformers too much, you can check it yourself.

[ftgreat]

Nice work.

[wwngh1233]

great job，the same with raw models on performance?

[qhduan]

great job，the same with raw models on performance?

Sorry, I have poor time to examine it. But check the generated response looks nice, I think even if it is different, the difference is very small

[sammysun0711]

Thanks for great work from @qhduan. Model can generate impressive outputs with HF AutoModelForCausalLM.

However, the model cannot be exported to ONNX because it contains ComplexFloat64 ops in Rotary Position Embedding (RoPE) implementation, which is not supported by ONNX export yet: https://github.com/pytorch/pytorch/issues/59246

Based on his work, I built another Hugging Face version AquliaChat-7B model, hope it will be useful for community. https://huggingface.co/sammysun0711/aquilachat-7b-hf

• Support Hugging Face Optimum: convert original RoPE implementation to Hugging Face RoPE implementation
• Support Inference with Hugging Face AutoModelForCausalLM, ORTModelForCausalLM, OVModelForCausalLM.

Reference: https://zhuanlan.zhihu.com/p/627536105

[ftgreat]

先关闭issue，如有问题重新打开，谢谢

[starlitsky2010]

@qhduan How did you convert from FlagAI model to huggingface model for the aquilachat model?

[sammysun0711]

Hi @starlitsky2010, I upload convert_aquila_weights_to_hf.py in my Hugging Face repo: https://huggingface.co/sammysun0711/aquilachat-7b-hf, will write a blog about model conversion, stay tuned.

[starlitsky2010]

Hi @starlitsky2010, I upload convert_aquila_weights_to_hf.py in my Hugging Face repo: https://huggingface.co/sammysun0711/aquilachat-7b-hf, will write a blog about model conversion, stay tuned.

Hi sammysun0711, the converting method is similar with Llama. I found that you've commented out the write_tokenizer part. How can we convert the tokenizer part? Any suggestions?

Thanks

[sammysun0711]

Hi @starlitsky2010, I upload convert_aquila_weights_to_hf.py in my Hugging Face repo: https://huggingface.co/sammysun0711/aquilachat-7b-hf, will write a blog about model conversion, stay tuned.

Hi sammysun0711, the converting method is similar with Llama. I found that you've commented out the write_tokenizer part. How can we convert the tokenizer part? Any suggestions?

Thanks

Hi @starlitsky2010, I think there is some difference between Aquila and LlaMA for Tokenizer,

• Original LlaMA use google Sentence Piece to train tokenizer.model (vocab_size = 32000)
• Aquila re-trained Tokenizer with Hugging Face GPT-NeoX tokenizer using extended vocabulary (vocab_size = 100008, including 8 special tokens), no tokenizer.model provided.

So for LlaMA, we need to convert tokenizer.model to Hugging Face LlaMA tokenizer, while for Aqulia model, we can direcly initialize Hugging Face GPT-NeoX tokenizer with configurations in tokenizer.json, vocab.json etc provided by Aquila.

@ftgreat, @qhduan, please feel free to correct me if I am wrong with something, thanks.

[sammysun0711]

Hi @starlitsky2010, I upgrade AquilaChat-7B v0.6 FP16 model in: https://huggingface.co/sammysun0711/aquilachat-7b-hf You may use following method to convert original pytorch model and tokenizer to Hugging Face format:

python convert_aquila_weights_to_hf.py --input_dir AquilaChat-7B --model_size 7B --output_dir aquilachat-7b-hf

Besides, I wrote a blog for AquilaChat-7B model enabling in Hugging Face: https://blog.openvino.ai/blog-posts/aquilachat-7b-language-model-enabling-with-hugging-face-optimum-intel