llCurious

Thanks for your replay. - So the normalization is applied in the first to ensure the inputs in the range [0,1]. Since you also mention that after quantization, rescaling is...

- **De-Normalization**: the input to `q_k` is firstly normalized into [-1,1] using non-linear transformation `weight = torch.tanh(self.weight) / torch.max(torch.abs(torch.tanh(self.weight)))`. Do you mean the de-quantization you mentioned [above](https://github.com/deJQK/AdaBits/blob/master/models/quant_ops.py#L149) is used to...

hi @hodlen , do you have any ideas?

Hi @Eric-mingjie . I try to benchmark the efficiency gain owing to the sparsity. However, i found that sparse matmul seems to be slower than dense matmul. ```Python sparsity_ratio =...

> Did you set the sparse kernel in `torch.sparse` as they did here https://pytorch.org/tutorials/prototype/semi_structured_sparse.html? > > ``` > import torch > from torch.sparse import to_sparse_semi_structured, SparseSemiStructuredTensor > from torch.utils.benchmark import...

@zhangwaer The related codes are: [experimental_mp](https://github.com/secretflow/spu/tree/ditto_icml24/examples/python/ml/experimental_mp) and [backend impl](https://github.com/secretflow/spu/blob/ditto_icml24/libspu/mpc/aby3/conversion.cc#L629). You can also run the [test script](https://github.com/secretflow/spu/blob/ditto_icml24/spu/tests/jnp_dynamic_ring_test.py) to have a try on dynamic ring computations.

Thx for your quick response! Regarding the practice > replicate this by modifying our code on the instruct pix2pix data do you mean runing the [Training CoDi on HuggingFace Data](https://github.com/fast-codi/CoDi?tab=readme-ov-file#training-codi-on-huggingface-data)...

Same question. Any ideas? @HaozheLiu-ST @WentianZhang-ML . Thx a lot!