MLOpsPython
MLOpsPython copied to clipboard
How to use azure devops artifacts repository as source for DatabricksStep of AzureML?
If we have PyPi Packages added as Artifacts to an Azure DevOps Project Feed, how can we use these packages as a source for installing packages in DatabricksStep of Azure Machine Learning Service?
While using pip in any environment, we use our Azure DevOps Project Artifacts feed in the following way:
pip install example-package --index-url=https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/
The DatabricksStep class of the Azure Machine Learning Service accepts the following parameters:
python_script_name = "<Some-Script>.py"
source_directory = "<Path-To-Script>"
<Some-Placeholder-Name-for-the-step> = DatabricksStep(
name=<Some-Placeholder-Name-for-the-step>,
num_workers=1,
python_script_name=python_script_name,
source_directory=source_directory,
run_name= <Name-of-the-run>,
compute_target=databricks_compute,
pypi_libraries = [
PyPiLibrary(package = 'scikit-learn'),
PyPiLibrary(package = 'scipy'),
PyPiLibrary(package = 'azureml-sdk'),
PyPiLibrary(package = 'joblib'),
PyPiLibrary(package = 'azureml-dataprep[pandas]'),
PyPiLibrary(package = 'example-package', repo='https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/')
],
allow_reuse=True
)
However, PyPiLibrary(package = 'example-package', repo='https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/') will give an error. How exactly should we consume the Artifacts Feed as an input to the PyPiLibrary property of the DatabricksStep Class in Azure Machine Learning Service?