Mlx generate text, by default halusonates more

[kishoretvk]

Issue Description Problem: When using the mix generate text command with verbose set to false, and the following parameters:

Temperature: 0.1 or 0 Top p: 1 The LLM models seem to hallucinate more frequently.

Models Used:

Mistral 7B v3 IBM Granite Mistral 7B v2 Use Case: Text-to-SQL

Method Used: mlx.generate_text

Steps to Reproduce Set up the environment with the specified models. Run the mix generate text command with verbose set to false. Use a temperature of 0.1 or 0 and top p of 1. Observe the output for hallucinations. Expected Behavior The models should generate accurate and coherent SQL queries without hallucinations.

Actual Behavior The models produce outputs that are factually incorrect or nonsensical, indicating hallucinations.

Additional Information Dataset: 20,000 examples split for training and validation. Hardware: 128GB M3 Max

[awni]

Could you share the commands you used and the precise model paths?

Hardware: 40 GB A100 GPU.

Also MLX is meant for Apple silicon.. is that a typo or were you running it on a linux machine?

[kishoretvk]

Sorry my bad 128 gb ram M3 max

[awni]

How about the commands / model paths?

Also when you say "by default halusonates more" what are you comparing against?