Support relative paths and regularize path handling and formatting

[AndydeCleyre]

This replaces #1329, rebased with care on top of pip-tools 6.8.0.

Changelog-friendly one-liners:

• Support relative paths in input files, as long as they lead with file:, <vcs>+file:, or ..
• Relative paths in input files become relative paths in output files.
• pip-compile will interpret relative paths in an input file as relative to that input file, rather than the current folder, if --read-relative-to-input is passed.
• pip-compile will reconstruct relative req paths as relative to the output file, rather than the current folder, if --write-relative-to-output is passed.
• pip-sync will interpret relative paths in an input file as relative to that input file, rather than the current folder, if --read-relative-to-input is passed.
• Annotation paths are now relative to the output file.
• Don't overwrite an input file ending in '.txt' when deriving the output file path.
• Don't confuse dots in folder names with file extensions when deriving the output file path.
• Include extras more reliably in output lines, like pkg[extra1,extra2].

---

• Fixes:
  • #204
  • #1107
  • #1165
• Related:
  • #453 (Not addressed in this PR)
  • #673
  • #702
  • #1067
  • pypa/pip#1289
  • pypa/pip#6112
  • pypa/pip#6247
  • pypa/pip#6276
  • pypa/pip#6658
  • pypa/pip#6905
  • pypa/pip#7650
  • pypa/pip#8035
  • pypa/pip#8765
  • pypa/pip#8828
  • pypa/packaging#264

Contributor checklist

• [x] Provided the tests for the changes.
• [ ] Assure PR title is short, clear, and good to be included in the user-oriented changelog

Maintainer checklist

• [ ] Assure one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
• [ ] Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[AndydeCleyre]

I think the Windows-only testing is not properly updating the coverage report.

[ivanst0]

@AndydeCleyre, the lines that are showing as not covered anymore after this PR in Codecov report are hit only on Python 3.11. There is only a single job running under Python 3.11 and it failed to upload the coverage to Codecov. Rerunning the checks should solve the problem.

Thank you for implementing this change!

[AndydeCleyre]

I'm just pushing periodic rebases, but I'm not making any changes.

[AndydeCleyre]

It seems the coverage upload fails very regularly. I wonder if anyone has interest in pursuing a change like this?

[webknjaz]

It seems the coverage upload fails very regularly. I wonder if anyone has interest in pursuing a change like this?

I don't think so, I keep using Codecov, and it worked well in the past years. There are bugs, of course, that get fixed (some — by me: https://github.com/codecov/uploader/issues?q=author%3Awebknjaz). But in any case, I'd first try to get to the root of the issue you're seeing before making such radical changes which may not improve things. Also, if we do this, I'd still keep Codecov uploads (even with a disabled check), because it provides a nice coverage view.

That said, it's clear that some jobs may have failed to upload coverage. It is currently hard to easily find out which ones. So I'd strongly recommend to:

1. Set up env-specific flags
2. Update the coverage action to v3 (it's currently v1 which is deprecated and uses the ancient uploader)
3. Audit the workflows for things that may need updating

re:3 — I've looked at https://github.com/jazzband/pip-tools/actions/runs/3175334504/jobs/5173249582#step:11:37 and was confused by it complaining about the actions/checkout action. So I scrolled up and, to my horror, saw it using the @master version. That branch has been deprecated in favor of @main over 2 years ago. Our setup currently uses the commit dated 13 Jul 2020 (HEAD of master). Having such a combination of old software versions could lead to the flakiness we're seeing now.

[allanlewis]

I've looked at https://github.com/jazzband/pip-tools/actions/runs/3175334504/jobs/5173249582#step:11:37 and was confused by it complaining about the actions/checkout action. So I scrolled up and, to my horror, saw it using the @master version. That branch has been deprecated in favor of @main over 2 years ago. Our setup currently uses the commit dated 13 Jul 2020 (HEAD of master). Having such a combination of old software versions could lead to the flakiness we're seeing now.

I noticed recently that Dependabot can now update your actions for you: https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot

[ssbarnea]

Please don't disable codecov, it fails because is primatly because it is not properly tuned. There are solutions for each of its problems. For example, making it require a number of success uploads bit lower than total. For example 9/10, or 8/10. Making it prevent failing a build if upload fails. Setting a rounding threshold to prevent it from posting a regression due to rounding errors.

It is essential to be able to browse the diff with its web interface in order to allow us to spot if some branches are not covered, especially for bigger changes.

Also, I would not keep its report as "required" for merge. I think we can all look at it an make a decision if we are about to ignore it or not.

[webknjaz]

I noticed recently that Dependabot can now update your actions for you: docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot

Yes, for as long as you're pointing at a tag or a commit sha but not branch. FWIW I've already made a PR updating this stuff. Along with a few others.

[webknjaz]

Also, I would not keep its report as "required" for merge. I think we can all look at it an make a decision if we are about to ignore it or not.

It was never required, just red.

[webknjaz]

FYI these GHA improvements are awaiting approvals: https://github.com/jazzband/pip-tools/pulls?q=is%3Apr+is%3Aopen+author%3Awebknjaz+sort%3Aupdated-desc

[ssbarnea]

Closing for the reasons mentioned by @atugushev - Please do not take it personally. I do not disagree with original idea but proposed implementation does touch too much code and adds undesired complexity. Maybe we could look into achieving this gradually with small/atomic changes, so we can slowly move the code-base towards the desired destination. Keep in mind that there are also other changes coming, so small changes will be less likely to conflict.

[AndydeCleyre]

Thank you for giving it as much chance as you did. All I can say is that I did my best given the constraints of backward compatibility and making no changes to pip itself.

[leifwalsh]

I would like to see something like this happen, as it causes extraneous diffs when updating requirements.txt files in automation that runs in temp directories. I'm currently working around it by patching over the generated files in my automation.

@ssbarnea any chance you could recommend how to break this up into smaller chunks? I totally feel where you're coming from and would be willing to help work on those but I don't know the codebase well enough to know how to split it up.

[AndydeCleyre]

FWIW these 29 commits are such that all tests pass at each step along the way, though not always fully covered by new tests at each step.