Interactive generation script

[younesbelkada]

Add small arguments that are accepted by accelerate for better performance in the previous script we were offloading to the disk which takes a lot of time

cc @Muennighoff

[Muennighoff]

I can't find any documentation on max_cpu_memory - Does this kwarg exist?

Traceback (most recent call last):
  File "generate.py", line 64, in <module>
    main()
  File "generate.py", line 41, in main
    model = AutoModelForCausalLM.from_pretrained(
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 446, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2070, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'max_cpu_memory'
bash-4.4$ pip show accelerate
Name: accelerate
Version: 0.11.0.dev0
Summary: Accelerate
Home-page: https://github.com/huggingface/accelerate
Author: The HuggingFace team
Author-email: [email protected]
License: Apache
Location: /gpfsssd/worksf/projects/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages
Requires: psutil, torch, packaging, pyyaml, numpy
Required-by:
bash-4.4$ pip show transformers
Name: transformers
Version: 4.21.0.dev0
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: [email protected]

[Muennighoff]

Also I'm pretty sure max_memory cannot be a string, but has to be a dictionary

[Muennighoff]

Just writing one line and CTRL+C (w/o Enter) yields the below for me. I think there is some batching issue.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "generate.py", line 64, in <module>
    main()
  File "generate.py", line 59, in main
    output = generate_from_text(model, text, tokenizer, max_length=args.generate_max_length, greedy=args.greedy, top_k=args.top_k)
  File "generate.py", line 25, in generate_from_text
    greedy_output = model.generate(
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/transformers/generation_utils.py", line 1288, in generate
    return self.greedy_search(
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/transformers/generation_utils.py", line 1683, in greedy_search
    outputs = self(
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/accelerate/hooks.py", line 148, in new_forward
    output = old_forward(*args, **kwargs)
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/transformers/models/bloom/modeling_bloom.py", line 821, in forward
    transformer_outputs = self.transformer(
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/accelerate/hooks.py", line 148, in new_forward
    output = old_forward(*args, **kwargs)
  File "/gpfswork/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/transformers/models/bloom/modeling_bloom.py", line 639, in forward
    input_ids = input_ids.view(-1, input_shape[-1])
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous

[Muennighoff]

Opened a PR with some changes: https://github.com/younesbelkada/bigscience/pull/1/