Ignore duplicate requirements

[q0w]

Bug description

With using pip-tools workflow for layered requirements pip-audit fails because of duplicate requirements.

Reproduction steps

echo "django" > a.in
pip-compile -q --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras a.in
echo -e "-c a.txt\ndjango-debug-toolbar" > b.in
pip-compile -q --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras b.in
pip-audit -r a.txt -r b.txt --require-hashes

Expected behavior

No errors

Screenshots and logs

DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='a.txt' mode='r' encoding='UTF-8'>, <_io.TextIOWrapper name='b.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=1, fix=False, require_hashes=True, index_url='https://pypi.org/simple/', extra_index_urls=[], skip_editable=False, no_deps=False, output=PosixPath('stdout'), ignore_vulns=[])
DEBUG:pip_audit._cli:Auditing asgiref (3.6.0)
DEBUG:pip_audit._cli:Auditing django (4.1.7)
DEBUG:pip_audit._cli:Auditing sqlparse (0.4.3)
ERROR:pip_audit._cli:package asgiref has duplicate requirements: asgiref==3.6.0 (from RequirementLine(line_number=7, line='asgiref==3.6.0     --hash=sha256:71e68008da809b957b7ee4b43dbccff33d1b23519fb8344e33f049897077afac     --hash=sha256:9567dfe7bd8d3c8c892227827c41cce860b368104c3431da67a0c5a65a949506', filename=PosixPath('/tmp/tmpifyxbix9')))

Platform information

• OS name and version: Arch Linux
• pip-audit version (pip-audit -V): pip-audit 2.5.2
• Python version (python -V or python3 -V): Python 3.10.10
• pip version (pip -V or pip3 -V): pip 23.0.1

Additional context

[woodruffw]

Thanks for the report @q0w!

This sounds like a bug -- could you please update your comment to use the bug report template? We use that to reduce the amount of manual maintainer effort needed to triage these kinds of things.

[q0w]

Updated

[woodruffw]

Much appreciated, thank you!

From a quick look, this stems from the current "fast path" behavior we have for --require-hashes. I believe it'll get resolved via #540; cc @tetsuo-cpp for opinions.

[tetsuo-cpp]

@woodruffw Yep that's correct, this should be fixed with #540.

[q0w]

@tetsuo-cpp pip-audit v2.5.3 fails now with --no-deps --fix. note: --no-deps is provided because of https://github.com/pypa/pip/issues/9644

[woodruffw]

@q0w could you provide the new error message?

[q0w]

I tried to make a repro, but no success. Also what should I do, if i wanna pip-audit to replace existing version rather than adding new line with fixed version. Its because of provided --no-deps?

WARNING:pip_audit._cli:--no-deps is supported, but users are encouraged to fully hash their pinned dependencies
WARNING:pip_audit._cli:Consider using a tool like `pip-compile`: https://pip-tools.readthedocs.io/en/latest/#using-hashes
WARNING:pip_audit._dependency_source.requirement:added fixed subdependency explicitly to requirements file main.txt: pyjwt
WARNING:pip_audit._dependency_source.requirement:encountered an exception while applying fixes, recovering original files: package pyjwt has duplicate requirements: pyjwt==2.4.0 (from RequirementLine(line_number=1575, line='pyjwt==2.4.0', filename='main.txt'))
Traceback (most recent call last):
  File "/tmp/e/venv/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/tmp/e/venv/lib/python3.10/site-packages/pip_audit/_cli.py", line 504, in audit
    source.fix(fix)
  File "/tmp/e/venv/lib/python3.10/site-packages/pip_audit/_dependency_source/requirement.py", line 168, in fix
    raise e
  File "/tmp/e/venv/lib/python3.10/site-packages/pip_audit/_dependency_source/requirement.py", line 162, in fix
    self._fix_file(filename, fix_version)
  File "/tmp/e/venv/lib/python3.10/site-packages/pip_audit/_dependency_source/requirement.py", line 189, in _fix_file
    raise RequirementFixError(
pip_audit._dependency_source.requirement.RequirementFixError: package pyjwt has duplicate requirements: pyjwt==2.4.0 (from RequirementLine(line_number=1575, line='pyjwt==2.4.0', filename='main.txt'))

[tetsuo-cpp]

@tetsuo-cpp pip-audit v2.5.3 fails now with --no-deps --fix. note: --no-deps is provided because of pypa/pip#9644

Hmm, that's because the --fix codepath doesn't use pip but instead parses the requirements file with pip-requirements-parser. I think we just need to remove this check since pip doesn't care about having multiple requirements for the same package.

Also what should I do, if i wanna pip-audit to replace existing version rather than adding new line with fixed version.

It should replace the existing version in the requirements file already. If it's adding a new line for a package that already exists in the requirements file, that's a bug. Do you have a repro?

[tetsuo-cpp]

I'm going to re-open this so we can address the --fix side of things too.

[q0w]

@tetsuo-cpp

echo "pyjwt==1.7.1" > requirements.in
pip-compile --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras
cat requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras
#
pyjwt==1.7.1 \
    --hash=sha256:5c6eca3c2940464d106b99ba83b00c6add741c9becaec087fb7ccdefea71350e \
    --hash=sha256:8d59a976fb773f3e6a39c85636357c4f0e242707394cadadd9814f5cbaa20e96
    # via -r requirements.in

pip-audit --fix --no-deps -r requirements.txt
cat requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras
#
pyjwt==1.7.1 \
    --hash=sha256:5c6eca3c2940464d106b99ba83b00c6add741c9becaec087fb7ccdefea71350e \
    --hash=sha256:8d59a976fb773f3e6a39c85636357c4f0e242707394cadadd9814f5cbaa20e96
# via -r requirements.in
    # pip-audit: subdependency explicitly fixed
pyjwt==2.4.0

It also fails without --no-deps, just --fix

[tetsuo-cpp]

Thanks @q0w. That's definitely a bug. I'll open a new issue for that.

The changeset at #577 should fix this.

[tetsuo-cpp]

@woodruffw --fix complaining about duplicates isn't new to 2.5.0 so this is probably not as urgent.

[q0w]

new case with pip-audit 2.6.0

echo docxcompose > a.in
echo gunicorn > b.in

pip-compile --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras a.in
pip-compile --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras b.in

pip-audit --fix --disable-pip -r a.txt -r b.txt

ERROR:pip_audit._cli:package setuptools has duplicate requirements: setuptools==68.0.0 (from RequirementLine(line_number=13, line='setuptools==68.0.0     --hash=sha256:11e52c67415a381d10d6b462ced9cfb97066179f0e871399e006c4ab101fc85f     --hash=sha256:baf1fdb41c6da4cd2eae722e135500da913332ab3f2f5c7d33af9b492acb5235', filename=PosixPath('b.txt')))

It also fails without --fix.

[q0w]

↑ This is a regression bug in 2.6.0 @woodruffw

[woodruffw]

↑ This is a regression bug in 2.6.0

Unless I'm misunderstanding, this is not a regression in 2.6.0: it's a known defect from the 2.4 and 2.5 releases. Are you saying that there's a new, different regression in 2.6.0, or are you confirming that the previous bug still exists?

[q0w]

This example above is working in <2.6.0

[woodruffw]

Gotcha, I understand now: I forgot that we left this open only for the --fix part, and that we resolved the duplicate requirement issue on audits. Sorry for the regression here; I'll look into a fix.