HT-NEKO

set languages=['en'], but contains results outside of English

1

Thank you for your work, it has been very helpful, but I have encountered some issues: my code: ``` ds = load_dataset( "/data/public/models/RedPajama-Data-V2/RedPajama-Data-V2/RedPajama-Data-V2.py", partition="head_middle", languages=["en"], name="sample",) ``` but ```ds``` contains...

训练过慢，且梯度为NaN

2

您好，感谢您的工作！我把您的clex layer部分插到我的模型中，实现方式如下： ``` class Encoder(nn.Module): def __init__(self, config): '''省略''' elif config.my_info_dict.get("algorithm",False)=="clex": from .clex_layer import CLEXScalingRotaryEmbedding rope_scaling={"factor": 1,"max_factor": 64,"param_factor": 1,"time_dt": 0.01,"type": "clex","act": "tanh"} self.clex_layer = CLEXScalingRotaryEmbedding(config.attention_key_size, self.config.my_info_dict["train_len"], rope_scaling) '''省略''' def forward(...