ctq

I also found it challenging to integrate the loglikelihood and generate functions into lm-evaluation-harness. If the author could open-source the relevant parts of lm-evaluation-harness, it would be greatly helpful to...

@Qubitium I also encountered the same problem. I believe that after packing a layer, deleting the corresponding FP32 fake quantized weights and releasing CPU memory could help when packing large...