`generate.py` and `generate/full.py` could be merged

[carmocca]

Proposed in https://github.com/Lightning-AI/lit-llama/pull/255

The only difference in logic is the instruction tuning. We could add a flag for it as in https://github.com/Lightning-AI/lit-llama/pull/278

[pavanpvl]

Hi @carmocca If possible I would like to work on this issue after going through the files and the discussions mentioned above what I can understand is as follows

1. Remove the generate.py file and add the required changes to the generate/full.py file
2. Add instruction tuning flag and use generate_prompt function if the flag is true
3. I can see that generate function in generate.py is referenced in multiple scripts(finetune, evaluate & generate) it can be moved to lit_llama/utils

Let me know if these changes sound good :D

[rasbt]

That's a good point. I also wondered if we even needed a generate.py script in the top level. Like you suggest, I am in favor of having the script under generate/ similar to the finetune/ and evaluate/ scripts.

I can see that generate function in generate.py is referenced in multiple scripts(finetune, evaluate & generate) it can be moved to lit_llama/utils

Or could we just reference the respective generate/* scripts then?

[pavanpvl]

@rasbt Yeah this sounds good as well, I was thinking of having it under utils but it might just be a matter of preference I guess 😅

[ruoyu61]

@carmocca I made a PR for this issue.

Could you help to review the PR?