uv icon indicating copy to clipboard operation
uv copied to clipboard

Update AWS CodeArtifact documentation to cover `[[tool.uv.index]]` API

Open AdrianB-sovo opened this issue 11 months ago • 5 comments

Currently, when using UV_INDEX and AWS CodeArtifact, a [[tool.uv.index]] section is automatically added in pyproject.toml (e.g. when calling uv add pandas) with the URL that includes the token, which mean I manually have to remove it before doing a git commit.

Here an example of lines generated in pyproject.toml when using AWS CodeArtifact:

[[tool.uv.index]]
url = "https://aws:<token>@somecompany-123456789012.d.codeartifact.eu-central-1.amazonaws.com/pypi/development/simple/"

Could there be a way to avoid that?

BTW: the doc https://docs.astral.sh/uv/guides/integration/alternative-indexes/#aws-codeartifact still uses the deprecated UV_EXTRA_INDEX_URL instead of UV_INDEX.

AdrianB-sovo avatar Dec 13 '24 14:12 AdrianB-sovo

UV can be configured to set secrets in environment variables, but I'm unsure if this approach is suitable for AWS. https://docs.astral.sh/uv/configuration/indexes/#providing-credentials

e.g.

[[tool.uv.index]]
name = "aws-code"
url = "https://somecompany-123456789012.d.codeartifact.eu-central-1.amazonaws.com/pypi/development/simple/"
export UV_INDEX_AWS_CODE_USERNAME=aws
export UV_INDEX_AWS_CODE_PASSWORD=<token>

I see that I've modified the twine section in this way. Perhaps I haven't made a mistake? https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-twine.html#python-configure-twine-without-login

[distutils]
index-servers =
 codeartifact
[codeartifact]
repository = https://my_domain-111122223333.d.codeartifact.us-west-2.amazonaws.com/pypi/my_repo/
password = auth-token
username = aws

FishAlchemist avatar Dec 13 '24 14:12 FishAlchemist

Thanks, it does work! :smiley: Maybe it should be the recommended method described in the documentation?

AdrianB-sovo avatar Dec 13 '24 15:12 AdrianB-sovo

Yeah the index integration docs were community contributed before we supported UV_INDEX — we should update those to cover both modes.

zanieb avatar Dec 13 '24 15:12 zanieb

@zanieb Should we update the password authentication part of Google Artifact Registry at the same time?

FishAlchemist avatar Dec 13 '24 15:12 FishAlchemist

Yeah it makes sense to update it all — I don't feel strongly about separate pull requests vs one.

zanieb avatar Dec 13 '24 15:12 zanieb

I was about to open an identical issue, but it seems that writing the index in [[tool.uv.index]] is intentional.

The workaround I'm currently using is to set it to UV_EXTRA_INDEX_URL:

export UV_EXTRA_INDEX_URL="https://aws:${AWS_CODEARTIFACT_TOKEN}@${AWS_DOMAIN}-${AWS_ACCOUNT_ID}.d.codeartifact.${AWS_REGION}.amazonaws.com/pypi/${AWS_CODEARTIFACT_REPOSITORY}/simple/"

When I run uv add <package>, it says:

$ uv add --dev "pytest>=8.3.4"
warning: Indexes specified via `--extra-index-url` will not be persisted to the `pyproject.toml` file; use `--index` instead.

which is exactly what I want, as UV_EXTRA_INDEX_URL contains a secret.

@FishAlchemist suggested a way to keep the index URL secret while using the new UV_INDEX configuration: https://docs.astral.sh/uv/configuration/indexes/#providing-credentials.

However, after reading the page, I'm unsure how to set it up with this method.

According to https://docs.astral.sh/uv/guides/integration/alternative-indexes/#aws-codeartifact I have:

  • AWS_DOMAIN
  • AWS_ACCOUNT_ID
  • AWS_ACCOUNT_ID
  • AWS_CODEARTIFACT_REPOSITORY
  • AWS_CODEARTIFACT_TOKEN

Where should I put each one of these, using the other method? It feels like I'm missing something trivial.

lmanc avatar Dec 23 '24 17:12 lmanc

I think I figured it out, maybe...?

[[tool.uv.index]]
name = "custom-name"
url = "https://<AWS_DOMAIN>-<AWS_ACCOUNT_ID>.d.codeartifact.<AWS_REGION>.amazonaws.com/pypi/<AWS_CODEARTIFACT_REPOSITORY>/simple/"

Where <> are placeholders, not environment variables.

Then, in .bashrc or wherever environment variables are defined:

export AWS_DOMAIN=
export AWS_ACCOUNT_ID=
export AWS_REGION=
export AWS_CODEARTIFACT_REPOSITORY=

export AWS_CODEARTIFACT_TOKEN="$(
    aws codeartifact get-authorization-token \
    --domain $AWS_DOMAIN \
    --domain-owner $AWS_ACCOUNT_ID \
    --region $AWS_REGION \
    --query authorizationToken \
    --output text
)"

export UV_INDEX_CUSTOM_NAME_USERNAME="aws"
export UV_INDEX_CUSTOM_NAME_PASSWORD="$AWS_CODEARTIFACT_TOKEN"

Where:

  • name = <custom-name> must match UV_INDEX_<CUSTOM_NAME>_USERNAME, written in all caps with _ replacing -
  • aws is just a placeholder for logging into CodeArtifact via a token.
  • UV_INDEX_<NAME>_PASSWORD could also be a token.

Am I right?

This is slightly counterintuitive, and I agree that it should be added to the documentation.

lmanc avatar Dec 23 '24 18:12 lmanc

@lmanc You seem to be on the right track, and this approach looks reasonable to me.


Here’s how I understand it could work:

Adding a Package with a Specific Index:

  1. Set Environment Variables:

    export AWS_DOMAIN=
    export AWS_ACCOUNT_ID=
    export AWS_REGION=
    export AWS_CODEARTIFACT_REPOSITORY=
    
    export AWS_CODEARTIFACT_TOKEN="$(
        aws codeartifact get-authorization-token \
        --domain $AWS_DOMAIN \
        --domain-owner $AWS_ACCOUNT_ID \
        --region $AWS_REGION \
        --query authorizationToken \
        --output text
    )"
    
    export UV_INDEX_CUSTOM_NAME_USERNAME="aws"
    export UV_INDEX_CUSTOM_NAME_PASSWORD="$AWS_CODEARTIFACT_TOKEN"
    
  2. Add a Package Using UV:

    uv add <PACKAGE_NAME> --index custom_name=<INDEX_URL>
    

Alternatively:

If you’d rather include the index URL directly, you might use a command like this:

uv add <PACKAGE_NAME> --index custom_name=https://${AWS_DOMAIN}-${AWS_ACCOUNT_ID}.d.codeartifact.${AWS_REGION}.amazonaws.com/pypi/${AWS_CODEARTIFACT_REPOSITORY}/simple/

This should add the following lines to your pyproject.toml file:

[tool.uv.sources]
<PACKAGE_NAME> = { index = "custom-name" }

[[tool.uv.index]]
name = "custom-name"
url = "<INDEX_URL>"

Using the Index Exclusively for a Single Library:

If you want the index to apply only to the specific library, you could try adding explicit = true like this:

[tool.uv.sources]
<PACKAGE_NAME> = { index = "custom-name" }

[[tool.uv.index]]
name = "custom-name"
url = "<INDEX_URL>"
explicit = true

If there’s anything I’ve misunderstood or missed, feel free to correct me. Also, I agree that adding this to the documentation would make things clearer for others encountering similar challenges. 😊

Note: <PACKAGE_NAME> and <INDEX_URL> are placeholders. Replace them with the actual package name and index URL.

Haknt avatar Jan 06 '25 13:01 Haknt

I still see an issue with that in my case, where we have dev and production indexes in CodeArtifact. Currently, we have:

pyproject.toml

# ...

dependencies = [
    "my-package[aws,utils]==3.0.123",
]

[[tool.uv.index]]
name = "aws-codeartifact"
url = "https://company-123456789.d.codeartifact.eu-central-1.amazonaws.com/pypi/development/simple/"

When we develop, we export the two following env variables:

export UV_INDEX_AWS_CODEARTIFACT_USERNAME=aws
export UV_INDEX_AWS_CODEARTIFACT_PASSWORD="${CODEARTIFACT_TOKEN}"

This will create a uv.lock with something like:

[[package]]
name = "my-package"
version = "3.0.123"
source = { registry = "https://company-123456789.d.codeartifact.eu-central-1.amazonaws.com/pypi/development/simple/" }
sdist = { url = "https://company-123456789.d.codeartifact.eu-central-1.amazonaws.com/pypi/development/simple/my-package/3.0.123/my-package-3.0.123.tar.gz", hash = "sha256:12345..." }
wheels = [
    { url = "https://company-123456789.d.codeartifact.eu-central-1.amazonaws.com/pypi/development/simple/my-package/3.0.123/my-package-3.0.123-py3-none-any.whl", hash = "sha256:12345..." },
]

Now, I'm searching how I could instead use the production index instead when deploying into production with the CI (i.e. https://company-123456789.d.codeartifact.eu-central-1.amazonaws.com/pypi/production/simple/).

One way would be to do uv export --frozen --no-dev --no-hashes > requirements.txt, then install using pip like I would usually do before using uv, but it's not a great solution...

AdrianB-sovo avatar Jan 06 '25 22:01 AdrianB-sovo

Started a draft to rework the documentation in https://github.com/astral-sh/uv/pull/10826.

The main reason it's in draft is because I won't be able to test AWS CodeArtifact nor Google Artifact Registry until next week, and won't be able to test Azure Artifacts at all. But if someone is able to test the changes before that, especially for Azure Artifacts as I can't, feel free to report what works and what doesn't in the PR.

mkniewallner avatar Jan 21 '25 21:01 mkniewallner