Text-to-code generation

[Chris33Edwards]

Thanks to your wonderful work. However, I wonder how can we use CodeT5+ to generate code based on the natural language description?

[rabbitjy]

same question

[yuewang-cuhk]

Hi there, please refer to the instructions here to reproduce the HumanEval results.

Below is a quick example to load the model and do the generation based on a given prompt:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch

checkpoint = "Salesforce/instructcodet5p-16b"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint,
                                              torch_dtype=torch.float16,
                                              low_cpu_mem_usage=True,
                                              trust_remote_code=True).to(device)

encoding = tokenizer("def print_hello_world():", return_tensors="pt").to(device)
encoding['decoder_input_ids'] = encoding['input_ids'].clone()
outputs = model.generate(**encoding, max_length=15)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

[Chris33Edwards]

Hi there, please refer to the instructions here to reproduce the HumanEval results.

Below is a quick example to load the model and do the generation based on a given prompt:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch

checkpoint = "Salesforce/instructcodet5p-16b"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint,
                                              torch_dtype=torch.float16,
                                              low_cpu_mem_usage=True,
                                              trust_remote_code=True).to(device)

encoding = tokenizer("def print_hello_world():", return_tensors="pt").to(device)
encoding['decoder_input_ids'] = encoding['input_ids'].clone()
outputs = model.generate(**encoding, max_length=15)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

I appreciate your share of the text-to-generation section, and I am wondering if there is a model with smaller parameters, such as 4b or 6b?

[Chris33Edwards]

Because I don't have enough video memory, so I set the value of device to 'cpu' and got this error. I hope you can help me solve this problem, thank you very much! Traceback (most recent call last): File "codegen.py", line 16, in outputs = model.generate(**encoding, max_length=15) File "/opt/conda/envs/acv/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context return func(*args, **kwargs) File "/opt/conda/envs/acv/lib/python3.7/site-packages/transformers/generation/utils.py", line 1269, in generate inputs_tensor, model_kwargs, model_input_name File "/opt/conda/envs/acv/lib/python3.7/site-packages/transformers/generation/utils.py", line 634, in _prepare_encoder_decoder_kwargs_for_generation model_kwargs["encoder_outputs"]: ModelOutput = encoder(**encoder_kwargs) File "/opt/conda/envs/acv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/root/.cache/huggingface/modules/transformers_modules/Salesforce/instructcodet5p-16b/70bb08afa3d6f081b347e67752ca8e031a35ac4a/modeling_codet5p.py", line 572, in forward output_attentions=output_attentions, File "/opt/conda/envs/acv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/root/.cache/huggingface/modules/transformers_modules/Salesforce/instructcodet5p-16b/70bb08afa3d6f081b347e67752ca8e031a35ac4a/modeling_codet5p.py", line 325, in forward hidden_states = self.ln_1(hidden_states) File "/opt/conda/envs/acv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/opt/conda/envs/acv/lib/python3.7/site-packages/torch/nn/modules/normalization.py", line 190, in forward input, self.normalized_shape, self.weight, self.bias, self.eps) File "/opt/conda/envs/acv/lib/python3.7/site-packages/torch/nn/functional.py", line 2347, in layer_norm return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled) RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

[Arjun1999]

Facing similar errors.

[yuewang-cuhk]

Yes, we also have smaller models and you may try smaller models for running on CPU. See below for all the models we've released:

• CodeT5+ 220M and 770M: codet5p-220m and codet5p-770m.
• CodeT5+ 220M and 770M that are further tuned on Python subset: codet5p-220m-py and codet5p-770m-py.
• CodeT5+ 2B, 6B, 16B: codet5p-2b, codet5p-6b, and codet5p-16b.
• InstructCodeT5+ 16B: instructcodet5p-16b.

[rodrigoandrigo]

Changing torch.float16 to torch.float32 might solve the problem I think

[ByteCopilot]

Changing torch.float16 to torch.float32 might solve the problem I think

it worked