Added an example Notebook to fine-tune Llama3 model using PyTorchJob

[aishwaryaraimule21]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged): Fixes #

Checklist:

• [ ] Docs included if any changes are user facing

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[Electronic-Waste]

I have no objections:)

[google-oss-prow[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign terrytangyuan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

Thank you for this effort @aishwaryaraimule21! I am fine with merging this KFP example. Any thoughts @johnugeorge @tenzen-y @Electronic-Waste @astefanutti ?

In that case, what are the relationship Training examples in KFP repository something like https://github.com/kubeflow/pipelines/tree/472f8779ded18f8904c5cbe15c0573d461d57af5/components/kubeflow/pytorch-launcher?

[andreyvelich]

Thank you for this effort @aishwaryaraimule21! I am fine with merging this KFP example. Any thoughts @johnugeorge @tenzen-y @Electronic-Waste @astefanutti ?

In that case, what are the relationship Training examples in KFP repository something like https://github.com/kubeflow/pipelines/tree/472f8779ded18f8904c5cbe15c0573d461d57af5/components/kubeflow/pytorch-launcher?

I think, you can use PyTorch launcher or you can directly use kubeflow-training SDK in the lightweight KFP component. It is up to the user to decide.

[tenzen-y]

Thank you for this effort @aishwaryaraimule21! I am fine with merging this KFP example. Any thoughts @johnugeorge @tenzen-y @Electronic-Waste @astefanutti ?

In that case, what are the relationship Training examples in KFP repository something like https://github.com/kubeflow/pipelines/tree/472f8779ded18f8904c5cbe15c0573d461d57af5/components/kubeflow/pytorch-launcher?

I think, you can use PyTorch launcher or you can directly use kubeflow-training SDK in the lightweight KFP component. It is up to the user to decide.

SGTM. It would be great if we could provide comprehensive examples after we release the consolidated SDK (I know the first version of SDK will be contained only katib and trainer features).

[andreyvelich]

@aishwaryaraimule21 Can you sign the DCO please ?

[aishwaryaraimule21]

@andreyvelich I have signed the DCO. Please check. Thanks.

[coveralls]

Pull Request Test Coverage Report for Build 13375853453

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 13314191840:	0.0%
Covered Lines:	85
Relevant Lines:	85

---

💛 - Coveralls

[andreyvelich]

Hi @aishwaryaraimule21, did you get a chance to address the @astefanutti feedback, so we can merge this example to the release-1.9 branch ?

[aishwaryaraimule21]

@andreyvelich I have tested the distributed training workflow using an older trainer image of release-1.9 branch. With the latest trainer package, I am running into an OOM error for the same TrainingArgs and hardware setup. I see the transformers package in the release-1.9 branch got upgraded from 4.38.0 to 4.50.2 during this time. https://github.com/kubeflow/trainer/commit/f58e8934dd77ed9334d92160c28e49d686f5c29f#diff-3bbef68e7a1f42b8d4d1ef6f0[…]9a1adeb7a3ec7e2a0f4d153c79276R3

[andreyvelich]

@andreyvelich I have tested the distributed training workflow using an older trainer image of release-1.9 branch. With the latest trainer package, I am running into an OOM error for the same TrainingArgs and hardware setup. I see the transformers package in the release-1.9 branch got upgraded from 4.38.0 to 4.50.2 during this time. f58e893#diff-3bbef68e7a1f42b8d4d1ef6f0%5B%E2%80%A6%5D9a1adeb7a3ec7e2a0f4d153c79276R3

Do you want to try to update other packages and try it again @aishwaryaraimule21 ?

[aishwaryaraimule21]

Do you want to try to update other packages and try it again @aishwaryaraimule21 ?

Yes, @andreyvelich. Let me try updating other packages. I tried running this example for smaller models like SmolLM2-135M-Instruct. The training is successful but they fail at the subsequent steps. I haven't gotten a chance to debug this. I will try to fix it this week. Thanks!