prefix tuning for Megatron gpt models

[arendu]

What does this PR do ?

implements prefix tuning for Megatron GPT models.

Collection: examples/nlp/languge_modeling

Changelog

• Add specific line by line info of high level changes in this PR.

Usage

/NeMo/examples/nlp/language_modeling$python megatron_gpt_prefix_tuning.py

The prefix tuning .yaml file is here: /NeMo/examples/nlp/language_modeling/conf/megatron_gpt_prefix_tuning_config.yaml, the settings in this config are used by default when NeMo/examples/nlp/language_modeling/megatron_gpt_prefix_tuning.py is launched.

Once trained, you can evaluate using:

python megatron_gpt_eval.py \
    prefix_tuned_model_file=<PATH_.nemo_FILE_GENERATED_AT_END_OF_TRAINING> \
    gpt_model_file=<PATH_TO_.nemo_FILE_WITH_PRETRAINED_GPT_MODEL> \
    inference.greedy=True \
    inference.add_BOS=False \
    trainer.devices=1 \
    trainer.num_nodes=1 \
    tensor_model_parallel_size=1 \
    pipeline_model_parallel_size=1 \
    prompts=["<PATH_TO_.jsonl_FILE_TO_EVALUATE>"]

Before your PR is "Ready for review"

Pre checks:

• [ ] Make sure you read and followed Contributor guidelines
• [ ] Did you write any new necessary tests?
• [ ] Did you add or update any necessary documentation?
• [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • [ ] Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• [x] New Feature
• [ ] Bugfix
• [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed. Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[lgtm-com[bot]]

This pull request introduces 15 alerts when merging b777ca8eadbef60471c135565b73724cfafc556c into 8abe0f4a71443950d4c82fd91f7eafc0800ac300 - view on LGTM.com

new alerts:

• 14 for Unused import
• 1 for Wrong name for an argument in a class instantiation

[lgtm-com[bot]]

This pull request introduces 15 alerts when merging 2405471a5fb2597480a8169937ccc1de61e824ac into fea3775c00adfacfe0a414dea15544abc96db8dc - view on LGTM.com

new alerts:

• 14 for Unused import
• 1 for Wrong name for an argument in a class instantiation

[lgtm-com[bot]]

This pull request introduces 15 alerts when merging 17a13b7f5552f292bfdbcd25cd04e2c18376ebc6 into fea3775c00adfacfe0a414dea15544abc96db8dc - view on LGTM.com

new alerts:

• 14 for Unused import
• 1 for Wrong name for an argument in a class instantiation

[lgtm-com[bot]]

This pull request introduces 15 alerts when merging 78762f2c5e281b4efd1d05cc242746aac34a2812 into fea3775c00adfacfe0a414dea15544abc96db8dc - view on LGTM.com

new alerts:

• 14 for Unused import
• 1 for Wrong name for an argument in a class instantiation

[lgtm-com[bot]]

This pull request introduces 15 alerts when merging 0c83654721c7f34390f49948f32dbafef9f24b34 into fea3775c00adfacfe0a414dea15544abc96db8dc - view on LGTM.com

new alerts:

• 14 for Unused import
• 1 for Wrong name for an argument in a class instantiation

[lgtm-com[bot]]

This pull request introduces 2 alerts when merging 5a95ec91fef66d6aad8e68017092ba1c42711dd2 into e67c4ca29a3855d1575f173af5b38ed3a9a91e68 - view on LGTM.com

new alerts:

• 1 for Unused import
• 1 for Wrong name for an argument in a class instantiation

[lgtm-com[bot]]

This pull request introduces 2 alerts when merging a132a48f8b6462d9ef81591a0a92e87d4675a77c into e67c4ca29a3855d1575f173af5b38ed3a9a91e68 - view on LGTM.com

new alerts:

• 1 for Unused import
• 1 for Wrong name for an argument in a class instantiation

[lgtm-com[bot]]

This pull request introduces 1 alert when merging c10282a56f55c9159f4e741f57ed823e4f385a17 into e67c4ca29a3855d1575f173af5b38ed3a9a91e68 - view on LGTM.com

new alerts:

• 1 for Wrong number of arguments in a class instantiation

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 4900d4d3bb9a2e3d8b0623d468d5d114afcb75f5 into e67c4ca29a3855d1575f173af5b38ed3a9a91e68 - view on LGTM.com

new alerts:

• 1 for Wrong number of arguments in a class instantiation

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 6f3f8ccea82455c31b428053319674c97eb106d7 into e67c4ca29a3855d1575f173af5b38ed3a9a91e68 - view on LGTM.com

new alerts:

• 1 for Wrong number of arguments in a class instantiation

[lgtm-com[bot]]

This pull request introduces 8 alerts when merging 3ea4336eca0ae78f98947e8ceb9abcd379352ebb into df335fe0ec110b0846521580734da447da06a24e - view on LGTM.com

new alerts:

• 4 for Unused import
• 2 for Unused local variable
• 1 for Unreachable code
• 1 for Wrong number of arguments in a class instantiation

[lgtm-com[bot]]

This pull request introduces 9 alerts when merging d455ac538c9fe8819334871ac0be4abe8dd92af3 into df335fe0ec110b0846521580734da447da06a24e - view on LGTM.com

new alerts:

• 6 for Unused import
• 2 for Unused local variable
• 1 for Wrong number of arguments in a class instantiation

[lgtm-com[bot]]

This pull request introduces 4 alerts when merging 266853bea82c11b28550a6dd59923c649743ca40 into 987674e29ea90f9a2f663bf95d74bd947d76bbc0 - view on LGTM.com

new alerts:

• 2 for Unused local variable
• 1 for Unused import
• 1 for Wrong number of arguments in a class instantiation

[lgtm-com[bot]]

This pull request introduces 4 alerts when merging da8e27d9198651a434cf2c4607783b3141fe9fb8 into f921ebe0436e55f7547b183ca83a623f6678422d - view on LGTM.com

new alerts:

• 2 for Unused local variable
• 1 for Unused import
• 1 for Wrong number of arguments in a class instantiation

[github-actions[bot]]

This PR is stale because it has been open for 30 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

[github-actions[bot]]

This PR was closed because it has been inactive for 7 days since being marked as stale.