MLOpsPython icon indicating copy to clipboard operation
MLOpsPython copied to clipboard

How to use azure devops artifacts repository as source for DatabricksStep of AzureML?

Open anirbansaha96 opened this issue 4 years ago • 0 comments
trafficstars

If we have PyPi Packages added as Artifacts to an Azure DevOps Project Feed, how can we use these packages as a source for installing packages in DatabricksStep of Azure Machine Learning Service?

While using pip in any environment, we use our Azure DevOps Project Artifacts feed in the following way:

pip install example-package --index-url=https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/

The DatabricksStep class of the Azure Machine Learning Service accepts the following parameters:

python_script_name = "<Some-Script>.py"
source_directory = "<Path-To-Script>"

<Some-Placeholder-Name-for-the-step> = DatabricksStep(
    name=<Some-Placeholder-Name-for-the-step>,
    num_workers=1,
    python_script_name=python_script_name,
    source_directory=source_directory,
    run_name= <Name-of-the-run>,
    compute_target=databricks_compute,
    pypi_libraries = [
                      PyPiLibrary(package = 'scikit-learn'), 
                      PyPiLibrary(package = 'scipy'), 
                      PyPiLibrary(package = 'azureml-sdk'), 
                      PyPiLibrary(package = 'joblib'), 
                      PyPiLibrary(package = 'azureml-dataprep[pandas]'),
                      PyPiLibrary(package = 'example-package', repo='https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/')
                    ], 

    allow_reuse=True
)

However, PyPiLibrary(package = 'example-package', repo='https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/') will give an error. How exactly should we consume the Artifacts Feed as an input to the PyPiLibrary property of the DatabricksStep Class in Azure Machine Learning Service?

anirbansaha96 avatar Jan 21 '21 14:01 anirbansaha96