[Usage] LLaVA outputted incoherent sentences on Arm arch

[y-vectorfield]

Describe the issue

Issue: LLaVA outptted incoherent sentences when we implemented on Arm architecure cpu.

Environment

• OS: "Rocky Linux 9.1
• CPU: Ampere Altra Max(128C128T)
• DRAM: 256GB
• GPU: NVIDA A100 PCIe 80GB

Softwares

• Docker 23.0.4(use nvcr.io/nvidia/pytorch:23.05-py3 image)
• CUDA 12.1
• Python 3.10.6
• accelerate 0.24.1
• huggingface-hub 0.20.1
• llava 1.1.3(built the latest version)
• safetensors 0.4.0
• sentencepiece 0.1.98
• torch 2.0.0
• torchvision 0.15.1
• transformers 4.31.0

Parameters

I set following parameters. (Other parametes used by defaults.)

• Model: liuhaotian/llava-v1.5-13b
• IMG: https://llava-vl.github.io/static/images/view.jpg
• Prompt: What are the things I should be cautious about when I visit here?
• Data Type: Float32
• Random Seed: 240119
• torch.use_deterministic_algorithms: True
• torch.backends.cudnn.deterministic: True
• torch.backends.cudnn.benchmark: False
• no_repeat_ngram_size: 2

When I implemented LLaVA using eval_model() function on above env, it outputted incoherent sentences like the following appear.(These contents was written in non-English texts that make no sense.) If I changed the random seed, it would only change the words repeated in the sentences.

When visiting the location, which is a wooden and arietially-paced, or a-fri-l-c-o-r, (a-t-b-a) or (c) and (b) (and) a (or) an (o) - and - a - (p) / (f) & (l) + a & a / a and the (w)e-n-e, a, & the, the & an, an & & and & 2 &2 and 1 and and all and some and many and one and two and three and four and five and six and seven and eight and nine and ten and eleven and twelve and thirteen and fifteen and twenty and thirty and forty and fifty and sietn andt ande and ea andi anda,a. and, all, some, many, one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirtie, 4,5,6,7,8,9,1,2,3,4 and so on.

For comparison, I also implemented this model on the following x86 env.

Environment

• OS: "Rocky Linux 8.7
• CPU: Intel Xeon Gold 6338T(24C48T)
• DRAM: 200GB
• GPU: NVIDIA RTX A6000 50GB

Softwares

• Docker 24.0.7(use nvcr.io/nvidia/pytorch:23.05-py3 image)
• CUDA 12.1
• Python 3.10.6
• accelerate 0.24.1
• huggingface-hub 0.20.1
• llava 1.1.3(built the latest version)
• safetensors 0.4.0
• sentencepiece 0.1.98
• torch 2.0.0
• torchvision 0.15.1
• transformers 4.31.0

Parameters

• Same condition on Arm arch

When visiting the pier over the lake, there are a few things to be mindful of for safety and enjoyment. First, be aware of the weather conditions, as the image shows a cloudy day. This could indicate potential rain or wind, which might make the experience less comfortable or even dangerous. Second, ensure that you have proper footwear, such as water shoes or boots, to prevent slipping on the wooden pier or getting wet from the water. Third, if you plan to swim or engage in water activities, make sure you are aware and follow any safety guidelines or rules posted at the location. Lastly, keep an eye on your belongings and be respectful to other visitors, ensuring a pleasant experience for everyone.