Add support for multiple requirements.txt files

[nateraw]

Describe the feature you'd like

Right now, its expected that requirements.txt will be found (With that name) in the script directory. In some cases, multiple requirements.txt files are needed to install the correct dependencies. Unfortunately, right now you'd have to edit your code in order to merge to one unified requirements.txt file if you have multiple.

How would this feature be used? Please describe.

Allow dependencies kwarg to pick up any .txt files as requirements files when they're added to the list.

estimator = HuggingFace(
    ...
    dependencies=['./requirements.txt', '../_test_requirements.txt']
)

Describe alternatives you've considered Right now, I'm:

• merging all requirements files a user specifies in a CLI tool I made
• Writing the merged requirements to a new tmp directory
• Adding all python files to the tmp directory as well
• Launching training runs out of the temp directory

I've also seen folks installing requirements directly in their python files, but that just feels wrong to me.

Additional context ...

[hesamparsa]

Is there any update on this thread? Is it possible to have multiple requirement files? thank you.