Credentials in extra index url are dropped

[notEvil]

Hi

Reproduction

1. set PROXPI_EXTRA_INDEX_URLS with username and password
2. use proxpi to fetch a package or fetch a file directly from /index/{package}/{file}

Expected behaviour

The package is downloaded.

Actual behaviour

proxpi drops the credentials and gets a http unauthorized response.

Environment

• proxpi: 1.2.0
• Environment: virtual environment
• Python: 3.12.7
• OS: Arch

I've tracked down the issue and the last change in this diff solves it:

diff --git a/proxpi/_cache.py b/proxpi/_cache.py
index 1d3ed1b..a1c2733 100644
--- a/proxpi/_cache.py
+++ b/proxpi/_cache.py
@@ -18,6 +18,8 @@ import urllib.parse
 import requests
 import lxml.etree
 
+import io
+
 INDEX_URL = os.environ.get("PROXPI_INDEX_URL", "https://pypi.org/simple/")
 EXTRA_INDEX_URLS = [
     s for s in os.environ.get("PROXPI_EXTRA_INDEX_URLS", "").strip().split(",") if s
@@ -401,7 +403,7 @@ class _IndexCache:
             )
             return
 
-        for _, child in lxml.etree.iterparse(response.raw, tag="a", html=True):
+        for _, child in lxml.etree.iterparse(io.BytesIO(response.text.encode()), tag="a", html=True):
             if True:  # minimise Git diff
                 name = _name_normalise_re.sub("-", child.text).lower()
                 self._index[name] = child.attrib["href"]
@@ -467,9 +469,9 @@ class _IndexCache:
             logger.debug(f"Finished listing files in package '{package_name}'")
             return
 
-        for _, child in lxml.etree.iterparse(response.raw, tag="a", html=True):
+        for _, child in lxml.etree.iterparse(io.BytesIO(response.text.encode()), tag="a", html=True):
             if True:  # minimise Git diff
-                file = FileFromHTML.from_html_element(child, response.request.url)
+                file = FileFromHTML.from_html_element(child, url)
                 package.files[file.name] = file
         self._packages[package_name] = package
         logger.debug(f"Finished listing files in package '{package_name}'")

The other changes fix encoding (maybe compression) issues which obfuscated this issue quite a bit. Let me know if I should open another issue for this. Relevant versions:

pypiserver 2.1.1 requests 2.32.3 lxml 5.3.0

[EpicWink]

I can't replicate. Could you please provide commands you ran to cause this issue?

My attempt:

docker run -p 5000:5000 -e PROXPI_EXTRA_INDEX_URLS epicwink/proxpi:v1.2.1rc0

pip download -vv --index-url http://127.0.0.1:5000/index <internal-package>
curl -v -L http://127.0.0.1:5000/index/<internal-package>/<internal-package>-1.0.0-py3-none-any.whl > /dev/null

[EpicWink]

Also, does #41 fix this for you?

[notEvil]

Hi, sry for not responding. The issue came up while evaluating different options and we eventually chose a different way which doesn't require a solution for it. At this point there is no test script/environment to recreate the issue easily, so feel free to close.