pipenv
pipenv copied to clipboard
Multi Platform support for Pipfile.lock
Is your feature request related to a problem? Please describe.
Currently it is not possible to create a Pipfile.lock for multiple platforms automatically. Only hashes for the current platform are locked. This is problematic when you want to use Pipefile.lock to share locked dependency versions in a multi platform setup, e.g. development on Windows or Mac and production on Linux.
E.g. cryptography generates the following Pipfile.lock entry on Windows 10 64bit:
"cryptography": {
"hashes": [
"sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
"sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
],
"markers": "python_version >= '3.6'",
"version": "==37.0.2"
},
and the following one on a Linux VM:
"cryptography": {
"hashes": [
"sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
"sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
"sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
"sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
],
"markers": "python_version >= '3.6'",
"version": "==37.0.2"
},
Describe the solution you'd like
My preferred solution would be a new entry in the Pipfile that defines which platforms hashes must always be included when locking. If the current systems platform isn't in the list it should be be added to the lock list though to retain backwards compatibility.
The Pipfile entry could look like this:
[[source]]
platforms = "manylinux_2_24_x86_64 or win_amd_64"
If possible it would be great if the values could be eagerly validated and an error message printed on invalid values.
Note: Ticket https://github.com/pypa/pipenv/issues/210 is similar to this one, but I was asked to open a new ticket
Could the lock file just be per-platform/version/etc?
Pipfile
defines my sources and packages.
- When I
pipenv lock
on my Python 3.8 on Windows 64bit, why couldn't it produce aPipfile.3.8.windows.x64.lock
? - If I try to
pipenv sync
on Python 3.10, since there is notPipfile.3.10.windows.x64.lock
, could the normal behavior kick in and a new lock would be generated? - Likewise, if I
pipenv lock
on Linux python 3.10, could it generate aPipfile.3.10.linux.x64.lock
?
What would be the drawback to that type of approach, aside from legacy support for the existing Pipfile.lock
?
What would the advantage of maintaining multiple lock files per platform and environment @dijital20 ? I am not a big fan of the idea of having platform specific lock files and then how to know which one to use during sync.
could the normal behavior kick in and a new lock would be generated?
Not for sync -- that would be very very bad because you loose the trust that you are verifying the dependencies you used in dev/test are the same you are deploying to production because the hashes would regenerate.
The current design can be extended to account for multiple platforms already -- for multi-platform dependencies all of the associated hashes are included in the lock, so this issue is really about dependencies that are for a different platform than the one you are locking on should also be included in this lock file with the relevant platform marker. Today this only happens if you lock on the environment that requirement is compatible with. Pipenv's own lock file has this problem -- if we do not lock on windows we loose a windows-specific dependency from the lock file.
EDIT: Just re-read the issue report and I am very surprised that the list of hashes for cryptography would be different between those two environments -- this part deserves some additional triage with the latest version of pipenv.
Yeah, platform-specific and Python-version-specific dependencies are the issue to solve here.
I had similar issues though when I moved a project from Python 3.8 to Python 3.10. My Pipfile and lock files both had Python 3.8 in them, and the standard pipenv sync --dev
failed unless I did pipenv sync --dev --python C:\python310\python.exe
, and then re-locked. At that point, everyone using Python 3.8 was out of luck, and so was anyone on Python 3.10.
I am not a big fan of the idea of having platform specific lock files and then how to know which one to use during sync.
pipenv
when called, would generate the lock file name based on the interpreter. In my examples, the names were Pipfile.{version}.{platform}.{architecture}.lock
. If the file exists, party on. If the file doesn't exist, then it should behave the same way pipenv
does today if the lock file doesn't exist.
I am not sure if platform-specific lock files is the solution... just thinking through it as we ran into this today, as a colleague is developing a new Python module that he wants to support both Windows and Linux on, but there are some dependencies which are platform specific.
Some alternatives to consider:
- Build the multi platform stuff into the lock file itself, but ideally in a better way than it is now.
- Add an option to
lock
,sync
,install
, anduninstall
to specify a lock file name/path. If not specified, use the default name today. This gives the user option to maintain separate specific lock files for different environments.
@dijital20 or @xyxz-web -- were either of you by chance using private python repositories isntead of pypi? I just locked on the latest pipenv main
on windows and cryptography has a lot more hashes -- like all of them -- than the original example had.
"cryptography": {
"hashes": [
"sha256:093cb351031656d3ee2f4fa1be579a8c69c754cf874206be1d4cf3b542042804",
"sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
"sha256:1b9362d34363f2c71b7853f6251219298124aa4cc2075ae2932e64c91a3e2717",
"sha256:1f3bfbd611db5cb58ca82f3deb35e83af34bb8cf06043fa61500157d50a70982",
"sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
"sha256:31fe38d14d2e5f787e0aecef831457da6cec68e0bb09a35835b0b44ae8b988fe",
"sha256:3b8398b3d0efc420e777c40c16764d6870bcef2eb383df9c6dbb9ffe12c64452",
"sha256:3c81599befb4d4f3d7648ed3217e00d21a9341a9a688ecdd615ff72ffbed7336",
"sha256:419c57d7b63f5ec38b1199a9521d77d7d1754eb97827bbb773162073ccd8c8d4",
"sha256:46f4c544f6557a2fefa7ac8ac7d1b17bf9b647bd20b16decc8fbcab7117fbc15",
"sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
"sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
"sha256:731c8abd27693323b348518ed0e0705713a36d79fdbd969ad968fbef0979a7e0",
"sha256:95e590dd70642eb2079d280420a888190aa040ad20f19ec8c6e097e38aa29e06",
"sha256:a68254dd88021f24a68b613d8c51d5c5e74d735878b9e32cc0adf19d1f10aaf9",
"sha256:a7d5137e556cc0ea418dca6186deabe9129cee318618eb1ffecbd35bee55ddc1",
"sha256:aeaba7b5e756ea52c8861c133c596afe93dd716cbcacae23b80bc238202dc023",
"sha256:dc26bb134452081859aa21d4990474ddb7e863aa39e60d1592800a8865a702de",
"sha256:e53258e69874a306fcecb88b7534d61820db8a98655662a3dd2ec7f1afd9132f",
"sha256:ef15c2df7656763b4ff20a9bc4381d8352e6640cfeb95c2972c38ef508e75181",
"sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e",
"sha256:f8ec91983e638a9bcd75b39f1396e5c0dc2330cbd9ce4accefe68717e6779e0a"
],
"index": "pypi",
"version": "==37.0.2"
},
For cryptography
, there's a compiled component written in Rust which is packaged along with the Python code. This would be different per-platform, which accounts for the different hashes per platform. This page on PyPI shows the different platform variations of the files:
https://pypi.org/project/cryptography/#files
The lock you posted @matteius looks like it got all of them. What @xyxz-web is seeing are probably the ones appropriate for the platform and interpreter they're using.
Like I said, I have a colleague who is trying to author a new package and wants to support both Windows and Linux, so he's developing out of both Windows and WSL2 Ubuntu. At least one of the dependencies (I suspect lxml) is different between the platforms, so if he locks on one platform, the lock file doesn't work on the other and vice versa.
We have changed to using lock
on Windows, and then lock --keep-outdated
on the Linux, but there's got to be an easier, slightly more obvious way.
I hope this is helpful. :) Thank you for the discussion on this.
@dijital20 You are right that locked that on windows, but I just tried on Ubuntu and got the same result:
"cryptography": {
"hashes": [
"sha256:093cb351031656d3ee2f4fa1be579a8c69c754cf874206be1d4cf3b542042804",
"sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
"sha256:1b9362d34363f2c71b7853f6251219298124aa4cc2075ae2932e64c91a3e2717",
"sha256:1f3bfbd611db5cb58ca82f3deb35e83af34bb8cf06043fa61500157d50a70982",
"sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
"sha256:31fe38d14d2e5f787e0aecef831457da6cec68e0bb09a35835b0b44ae8b988fe",
"sha256:3b8398b3d0efc420e777c40c16764d6870bcef2eb383df9c6dbb9ffe12c64452",
"sha256:3c81599befb4d4f3d7648ed3217e00d21a9341a9a688ecdd615ff72ffbed7336",
"sha256:419c57d7b63f5ec38b1199a9521d77d7d1754eb97827bbb773162073ccd8c8d4",
"sha256:46f4c544f6557a2fefa7ac8ac7d1b17bf9b647bd20b16decc8fbcab7117fbc15",
"sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
"sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
"sha256:731c8abd27693323b348518ed0e0705713a36d79fdbd969ad968fbef0979a7e0",
"sha256:95e590dd70642eb2079d280420a888190aa040ad20f19ec8c6e097e38aa29e06",
"sha256:a68254dd88021f24a68b613d8c51d5c5e74d735878b9e32cc0adf19d1f10aaf9",
"sha256:a7d5137e556cc0ea418dca6186deabe9129cee318618eb1ffecbd35bee55ddc1",
"sha256:aeaba7b5e756ea52c8861c133c596afe93dd716cbcacae23b80bc238202dc023",
"sha256:dc26bb134452081859aa21d4990474ddb7e863aa39e60d1592800a8865a702de",
"sha256:e53258e69874a306fcecb88b7534d61820db8a98655662a3dd2ec7f1afd9132f",
"sha256:ef15c2df7656763b4ff20a9bc4381d8352e6640cfeb95c2972c38ef508e75181",
"sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e",
"sha256:f8ec91983e638a9bcd75b39f1396e5c0dc2330cbd9ce4accefe68717e6779e0a"
],
"index": "pypi",
"version": "==37.0.2"
},
I am going to need y'all to provide the output from pipenv --support
to further debug,
I also get the long list of hashes when I lock from within WSL - seems to include both Windows and Linux hashes. However, when I lock from within Windows 10 or a Linux VM, I get a much shorter list. I'm usinge a private repo here, but it is automatically mirrored from pypi.
@xyxz-web hash collection of all packages is a specific feature of pypi. There is no API into private pypi servers for collecting the set of hashes, and I suspect that is where the problem comes in. For reference, here is the code that collects the hashes: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L757-L788 Specifically look for where it collects hashes from pypi: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L726-L755
I would recommend if possible to use pypi as a primary index in the Pipfile
and support your private packages as a secondary index calling out specifically the private index to pull those specific packages from.
how to know which one to use during sync
@matteius perhaps with:
import platform
# somwehere inside the `sync` method definition
open(f"Pipfile.lock.{platform.machine()}")
I would not include the Python version in the lockfile name, project should be developed for the version running on the production servers to avoid surprises and provide more or less stable development experience and features provided by the specific version of the language used.
I think this is a really important problem to solve right now. For example, if we have a project that uses numpy
as a dependency and we have an x86_64
server, Github Actions CI also on x86_64
, 1 developer on Linux x86_64
, one on older x86_64 Mac and one more with an M1 Macbook arm64. Each time someone updates the lockfile, at least one person has problem. At the moment, one quick solution I found was to manually modify the Pipfile.lock file and add this (for numpy):
"markers": "(platform_machine == 'x86_64' or platform_machine == 'arm64') and python_version < '3.10'",
in
"numpy": {
"hashes": [...],
"markers": "...",
"version": "==1.23.2"
(the above allows me to work on M1 Mac and run CI on x86_64 Github Action + deploy to amd64 Linux server, but it's not sustainable as next time someone adds or updates a package it'll be lost)
The Ruby Bundler solves this problem by adding separate lock file entries for each platform, e.g.
nokogiri (1.13.6)
nokogiri (1.13.6-aarch64-linux)
nokogiri (1.13.6-x86_64-darwin)
nokogiri (1.13.6-x86_64-linux)
I think this is a really important problem to solve right now.
I agree with you, it is becoming higher on the list of priorities and we have tackled a number of other bugs and technical debt items that set us up better to work on this. I am inclined to try and find a solution that will work with the single Pipfile.lock
than rather try to support multiple lock files by platform. I am not quite sure how it would work though for something that has to be built on that system architecture in order to generate the appropriate hashes because how could it build all of the architectures at once? That is probably where the thinking to have a separate platform specific lockfiles comes into play but I need to give this problem more time and consideration as well as additional inputs will be helpful.
@januszm Can you explain more about your workaround "markers": "(platform_machine == 'x86_64' or platform_machine == 'arm64') and python_version < '3.10'",
-- can you provide more context, maybe the whole block from that part of the lockfile so I can get a better idea of what the workaround is doing.
Another thing I have been thinking more about as it relates to pre-built wheels and private pypi's, is it would be great if we could get the private pypi servers to support a similar json API to pypi for fetching the available package hashes. Of the data that API returns, pipenv specifically leverages the part of the payload that is: api_releases = r.json()["releases"] to have the versions as keys with each having a key ["digests"]["sha256"] which would be the package hash for that version package. If we knew that more private pypi servers supported this informational package API, then we could update our logic here to try this for more than just pypi.org: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L727-L755
That wouldn't help for packages that need to be built on system architectures to get those hashes (because no pre-built wheels exist) but it would help in general with this issue and with performance, though a platform specific build solution I believe needs to exist as well to manage hashes generated from different lock architectures. I think the numpy issue though, there are pre-built wheels for most architectures and pipenv will not pull all of the hashes from a private pypi because it would have to download all of the wheels because such an API is not supported. Perhaps part of the solution could be an option to override and tell it to download all platform eligible wheels from a private pypi in order to collect all the hashes, but again I would need to study that code some more to find out.
Thanks for the detailed response @matteius . I don't know if I will be able to professionally explain what exactly this workaround is because I have never studied pip
and pipenv
internals, I am rather just a user of these systems. As I understand it, this line, adding or platform_machine == 'arm64'
just lets pip build a local extension for numpy, it unlocks the installation process on arm64 platform.
I am not sure how hashes work in pip and pipenv but IMO it seems unnecessary to calculate hashes for extensions that are built locally. The hashes for the source code, which is common to all architectures, should provide necessary minimum security so it can be computed once. Probably that's why it works with my workaround, the hash of files that are built locally doesn't matter because they come from source code that already has hashes.
UPDATE: I just received this from a friend who uses poetry to manage dependencies in the project. Looks like poetry solves this issue by storing a list of specifications for each platform for a particular dependency:
[package.dependencies]
numpy = [
{version = ">=1.18.5", markers = "platform_machine != \"aarch64\" and platform_machine != \"arm64\" and python_version < \"3.10\""},
{version = ">=1.19.2", markers = "platform_machine == \"aarch64\" and python_version < \"3.10\""},
{version = ">=1.20.0", markers = "platform_machine == \"arm64\" and python_version < \"3.10\""},
{version = ">=1.21.0", markers = "python_version >= \"3.10\""},
]
python-dateutil = ">=2.8.1"
# ...
@januszm Thanks I will take a closer look at that later, fwiw, I think you may be actually installing pre-built wheels for the platform. For example, there are prebuilt arm64 wheels for macos: https://pypi.org/project/numpy/1.23.2/#files
The issue for pipenv though for pre-built wheels is when going through a private pypi, I believe it does not download all of the wheels to get the hashes, only the ones that match the system you are locking on.
I think you may be actually installing pre-built wheels for the platform
Ah true, if we could make pipenv work with all platforms, not only the one we're locking on, it would already be a great improvement
TL;DR another case where this issue has a negative impact is when trying to publish a Jupyter notebook using mybinder
.
Details: I have a notebook that I develop locally (M1 mac) and I'm trying to serve it using https://mybinder.org/. During the build it seems like numpy
was not installed due to the platform specified in the lock file. In turn, the served notebook fails... See this discussion for more details.
Possible workaround: In https://github.com/drorata/candy-analysis/commit/ed2c6eb46f1fb740a0efee3d208ebe92dcc5aeef I removed the Pipfile.lock
so when the image will be built on mybinder
's side, it will be free to pick the appropriate platform. This unblocked my build and running of the notebook inside mybinder
.
One addition, FWIW: I now faced a similar related issue. I'm trying to dockerize some work which was developed on an M1 mac. To that end, in the Dockerfile
I added:
COPY ./Pipfile* .
RUN python -m pip install --upgrade pip
RUN pip install pipenv && pipenv install --dev --system --deploy
It comes as no surprise, that I get the following warning: Ignoring appnope: markers 'platform_system == "Darwin"' don't match your environment
. I tried not to copy the Pipfile.lock
, but then it is not possible to use the --system
flag when installing the environment. I don't think it harmed the running of the container but I find it very confusing.
@drorata I am not sure how the markers are making it into your Pipfile for numpy ... I just locked on many different systems numpy, on windows, linux VM, Mac M1, and all did not have the markers restricting numpy. What version of pipenv
are you on?
Also noting to the larger group that I am unable to reproduce getting not the full set of hashes for some of these packages on pipenv==2022.8.24
-- I tried multiple systems and I also tried pypiserver
which mirrors pypi packages, and in all cases the lock file has the full set of hashes. It appears even when it has to download the wheels, it downloads all of the matching version to get the hashes. If anyone can provide more reproduction steps with the current version of pipenv I would like to look more into it.
@matteius Thanks for your answer! I'm used version 2022.8.17
(installed using brew
). I upgraded it (to 2022.8.24
) and now there are no markers in the Pipfile.lock
. In turn the warning doesn't show up.
Does it mean this issue was fixed in 24.08?
EDIT: looks like it does, after the pipenv upgrade I see that the platform specific markers are gone. Example for numpy:
"markers": "python_version < '3.10'",
"version": "==1.23.2"
Does it mean this issue was fixed in 24.08?
That's a bold statement @januszm. I'm not sure what's meant by "this issue". In my case, using the newer version yielded a Pipfile.lock
file which didn't contain the "markers" and I was able to build the docker image without warnings.
I think it really needs to be one lockfile per platform because of stuff like: https://github.com/pandas-dev/pandas/blob/v1.4.3/setup.cfg#L33, which each platform has a different version requirement. Otherwise locks would have to munge a single lockfile which could prove problematic due to competing requirements
I think it's worth looking at the poetry file format for this type of scenario:
[package.dependencies]
numpy = [
{version = ">=1.18.5", markers = "platform_machine != \"aarch64\" and platform_machine != \"arm64\" and python_version < \"3.10\""},
{version = ">=1.19.2", markers = "platform_machine == \"aarch64\" and python_version < \"3.10\""},
{version = ">=1.20.0", markers = "platform_machine == \"arm64\" and python_version < \"3.10\""},
]
along with
numpy = [
{file = "numpy-1.23.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:aeba539285dcf0a1ba755945865ec61240ede5432df41d6e29fab305f4384db2"},
{file = "numpy-1.23.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:7e8229f3687cdadba2c4faef39204feb51ef7c1a9b669247d49a24f3e2e1617c"},
{file = "numpy-1.23.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1408c3527a74a0209c781ac82bde2182b0f0bf54dea6e6a363fe0cc4488a7ce7"},
{file = "numpy-1.23.1.tar.gz", hash = "sha256:d748ef349bfef2e1194b59da37ed5a29c19ea8d7e6342019921ba2ba4fd8b624"},
]
that way multiple versions are supported, with each platform getting what it requires
@thehesiod That is really complicated, I mean perhaps possible to specify all that to the pip resolver, but never the less couldn't your example be boiled down to this for pipenv
locking and work properly?
numpy = {version = ">=1.20.0", markers = "python_version < '3.10'"}
Since if you specify all three constraints for version to the resolver, you are going to have to get one version that is compatible across all constraints, so you may as well go with version = ">=1.20.0"
and then I guess the question becomes what are you trying to accomplish with all those marker constarints? It is not clear from the example, but the above line resolves a lock file entry like:
"numpy": {
"hashes": [
"sha256:17e5226674f6ea79e14e3b91bfbc153fdf3ac13f5cc54ee7bc8fdbe820a32da0",
"sha256:2bd879d3ca4b6f39b7770829f73278b7c5e248c91d538aab1e506c628353e47f",
"sha256:4f41f5bf20d9a521f8cab3a34557cd77b6f205ab2116651f12959714494268b0",
"sha256:5593f67e66dea4e237f5af998d31a43e447786b2154ba1ad833676c788f37cde",
"sha256:5e28cd64624dc2354a349152599e55308eb6ca95a13ce6a7d5679ebff2962913",
"sha256:633679a472934b1c20a12ed0c9a6c9eb167fbb4cb89031939bfd03dd9dbc62b8",
"sha256:806970e69106556d1dd200e26647e9bee5e2b3f1814f9da104a943e8d548ca38",
"sha256:806cc25d5c43e240db709875e947076b2826f47c2c340a5a2f36da5bb10c58d6",
"sha256:8247f01c4721479e482cc2f9f7d973f3f47810cbc8c65e38fd1bbd3141cc9842",
"sha256:8ebf7e194b89bc66b78475bd3624d92980fca4e5bb86dda08d677d786fefc414",
"sha256:8ecb818231afe5f0f568c81f12ce50f2b828ff2b27487520d85eb44c71313b9e",
"sha256:8f9d84a24889ebb4c641a9b99e54adb8cab50972f0166a3abc14c3b93163f074",
"sha256:909c56c4d4341ec8315291a105169d8aae732cfb4c250fbc375a1efb7a844f8f",
"sha256:9b83d48e464f393d46e8dd8171687394d39bc5abfe2978896b77dc2604e8635d",
"sha256:ac987b35df8c2a2eab495ee206658117e9ce867acf3ccb376a19e83070e69418",
"sha256:b78d00e48261fbbd04aa0d7427cf78d18401ee0abd89c7559bbf422e5b1c7d01",
"sha256:b8b97a8a87cadcd3f94659b4ef6ec056261fa1e1c3317f4193ac231d4df70215",
"sha256:bd5b7ccae24e3d8501ee5563e82febc1771e73bd268eef82a1e8d2b4d556ae66",
"sha256:bdc02c0235b261925102b1bd586579b7158e9d0d07ecb61148a1799214a4afd5",
"sha256:be6b350dfbc7f708d9d853663772a9310783ea58f6035eec649fb9c4371b5389",
"sha256:c403c81bb8ffb1c993d0165a11493fd4bf1353d258f6997b3ee288b0a48fce77",
"sha256:cf8c6aed12a935abf2e290860af8e77b26a042eb7f2582ff83dc7ed5f963340c",
"sha256:d98addfd3c8728ee8b2c49126f3c44c703e2b005d4a95998e2167af176a9e722",
"sha256:dc76bca1ca98f4b122114435f83f1fcf3c0fe48e4e6f660e07996abf2f53903c",
"sha256:dec198619b7dbd6db58603cd256e092bcadef22a796f778bf87f8592b468441d",
"sha256:df28dda02c9328e122661f399f7655cdcbcf22ea42daa3650a26bce08a187450",
"sha256:e603ca1fb47b913942f3e660a15e55a9ebca906857edfea476ae5f0fe9b457d5",
"sha256:ecfdd68d334a6b97472ed032b5b37a30d8217c097acfff15e8452c710e775524"
],
"markers": "python_version < '3.10'",
"version": "==1.23.2"
},
This includes all of the relevant hashes for this version. Is there a platform you are trying to exclude numpy from?
I spent some time on this today and what I found was somewhat discouraging -- internal to the pip resolver the markers of transitive dependencies are processed and those requirements are discarded if they are not for your platform. Pipenv has such a thing in its lock file: it depends on click which has metadata:
Requires-Dist: colorama ; platform_system == "Windows"
Requires-Dist: importlib-metadata ; python_version < "3.8"
So if we do not lock on Windows, the colorama is dropped from the lock file. The problem with my code modifications had nothing to do with colorama, but other dependencies that did not have wheell/metadata for my system and so it tries to build that metadata but fails to do so because the marker was ignored that indicates it was not for my system. I tried to hack around it, but I ended up in long locking phases that never ended, or ones that ended because some dependencie's possible extra could not generate the metadata on my system.
The interesting thing though, is by adding a * resolution of that same package to the Pipfile
, then the lock does work on linux and does not drop the colorama. For example adding: colorama = {version = "*", markers="sys_platform == 'win32'"}
to the Pipfile
of pipenv and locking on linux does not change the coloarama
in the lock file. I note this as a workaround to a fairly complex problem that I don't see an easy solution for. It is interesting though, requirementslib has a setup.py reader that generates some kind of metadata without invoking any of the setup.py logic, which seems quite different from what the pip resolver is doing--the pip resolver I believe is designed to prep packages for install and so perhaps it is reusing logic for getting the metadata that assumes the package will definitely be installed on that system, when what we want is to be able to resolve a proper lockfile that includes all platforms with the relevant markers.
I am thinking that named categories may help out with this issue -- See: https://github.com/pypa/pipenv/pull/5366
For example:
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
requests = "*"
[dev-packages]
[requires]
python_version = "3.10"
[linux]
numpy = "*"
[windows]
numpy = "==1.22.4"
matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/multi-platform
$ cat Pipfile.lock
{
"_meta": {
"hash": {
"sha256": "18815ba6d0514cb0588df0a7bc63bca22ea99223fc0ba15281b0258dbefff304"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.10"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {},
"develop": {},
"linux": {
"numpy": {
"hashes": [
"sha256:004f0efcb2fe1c0bd6ae1fcfc69cc8b6bf2407e0f18be308612007a0762b4089",
"sha256:09f6b7bdffe57fc61d869a22f506049825d707b288039d30f26a0d0d8ea05164",
"sha256:0ea3f98a0ffce3f8f57675eb9119f3f4edb81888b6874bc1953f91e0b1d4f440",
"sha256:17c0e467ade9bda685d5ac7f5fa729d8d3e76b23195471adae2d6a6941bd2c18",
"sha256:1f27b5322ac4067e67c8f9378b41c746d8feac8bdd0e0ffede5324667b8a075c",
"sha256:22d43376ee0acd547f3149b9ec12eec2f0ca4a6ab2f61753c5b29bb3e795ac4d",
"sha256:2ad3ec9a748a8943e6eb4358201f7e1c12ede35f510b1a2221b70af4bb64295c",
"sha256:301c00cf5e60e08e04d842fc47df641d4a181e651c7135c50dc2762ffe293dbd",
"sha256:39a664e3d26ea854211867d20ebcc8023257c1800ae89773cbba9f9e97bae036",
"sha256:51bf49c0cd1d52be0a240aa66f3458afc4b95d8993d2d04f0d91fa60c10af6cd",
"sha256:78a63d2df1d947bd9d1b11d35564c2f9e4b57898aae4626638056ec1a231c40c",
"sha256:7cd1328e5bdf0dee621912f5833648e2daca72e3839ec1d6695e91089625f0b4",
"sha256:8355fc10fd33a5a70981a5b8a0de51d10af3688d7a9e4a34fcc8fa0d7467bb7f",
"sha256:8c79d7cf86d049d0c5089231a5bcd31edb03555bd93d81a16870aa98c6cfb79d",
"sha256:91b8d6768a75247026e951dce3b2aac79dc7e78622fc148329135ba189813584",
"sha256:94c15ca4e52671a59219146ff584488907b1f9b3fc232622b47e2cf832e94fb8",
"sha256:98dcbc02e39b1658dc4b4508442a560fe3ca5ca0d989f0df062534e5ca3a5c1a",
"sha256:a64403f634e5ffdcd85e0b12c08f04b3080d3e840aef118721021f9b48fc1460",
"sha256:bc6e8da415f359b578b00bcfb1d08411c96e9a97f9e6c7adada554a0812a6cc6",
"sha256:bdc9febce3e68b697d931941b263c59e0c74e8f18861f4064c1f712562903411",
"sha256:c1ba66c48b19cc9c2975c0d354f24058888cdc674bebadceb3cdc9ec403fb5d1",
"sha256:c9f707b5bb73bf277d812ded9896f9512a43edff72712f31667d0a8c2f8e71ee",
"sha256:d5422d6a1ea9b15577a9432e26608c73a78faf0b9039437b075cf322c92e98e7",
"sha256:e5d5420053bbb3dd64c30e58f9363d7a9c27444c3648e61460c1237f9ec3fa14",
"sha256:e868b0389c5ccfc092031a861d4e158ea164d8b7fdbb10e3b5689b4fc6498df6",
"sha256:efd9d3abe5774404becdb0748178b48a218f1d8c44e0375475732211ea47c67e",
"sha256:f8c02ec3c4c4fcb718fdf89a6c6f709b14949408e8cf2a2be5bfa9c49548fd85",
"sha256:ffcf105ecdd9396e05a8e58e81faaaf34d3f9875f137c7372450baa5d77c9a54"
],
"index": "pypi",
"version": "==1.23.3"
}
},
"windows": {
"numpy": {
"hashes": [
"sha256:0791fbd1e43bf74b3502133207e378901272f3c156c4df4954cad833b1380207",
"sha256:1ce7ab2053e36c0a71e7a13a7475bd3b1f54750b4b433adc96313e127b870887",
"sha256:2d487e06ecbf1dc2f18e7efce82ded4f705f4bd0cd02677ffccfb39e5c284c7e",
"sha256:37431a77ceb9307c28382c9773da9f306435135fae6b80b62a11c53cfedd8802",
"sha256:3e1ffa4748168e1cc8d3cde93f006fe92b5421396221a02f2274aab6ac83b077",
"sha256:425b390e4619f58d8526b3dcf656dde069133ae5c240229821f01b5f44ea07af",
"sha256:43a8ca7391b626b4c4fe20aefe79fec683279e31e7c79716863b4b25021e0e74",
"sha256:4c6036521f11a731ce0648f10c18ae66d7143865f19f7299943c985cdc95afb5",
"sha256:59d55e634968b8f77d3fd674a3cf0b96e85147cd6556ec64ade018f27e9479e1",
"sha256:64f56fc53a2d18b1924abd15745e30d82a5782b2cab3429aceecc6875bd5add0",
"sha256:7228ad13744f63575b3a972d7ee4fd61815b2879998e70930d4ccf9ec721dce0",
"sha256:9ce7df0abeabe7fbd8ccbf343dc0db72f68549856b863ae3dd580255d009648e",
"sha256:a911e317e8c826ea632205e63ed8507e0dc877dcdc49744584dfc363df9ca08c",
"sha256:b89bf9b94b3d624e7bb480344e91f68c1c6c75f026ed6755955117de00917a7c",
"sha256:ba9ead61dfb5d971d77b6c131a9dbee62294a932bf6a356e48c75ae684e635b3",
"sha256:c1d937820db6e43bec43e8d016b9b3165dcb42892ea9f106c70fb13d430ffe72",
"sha256:cc7f00008eb7d3f2489fca6f334ec19ca63e31371be28fd5dad955b16ec285bd",
"sha256:d4c5d5eb2ec8da0b4f50c9a843393971f31f1d60be87e0fb0917a49133d257d6",
"sha256:e96d7f3096a36c8754207ab89d4b3282ba7b49ea140e4973591852c77d09eb76",
"sha256:f0725df166cf4785c0bc4cbfb320203182b1ecd30fee6e541c8752a92df6aa32",
"sha256:f3eb268dbd5cfaffd9448113539e44e2dd1c5ca9ce25576f7c04a5453edc26fa",
"sha256:fb7a980c81dd932381f8228a426df8aeb70d59bbcda2af075b627bbc50207cba"
],
"index": "pypi",
"version": "==1.22.4"
}
}
}
Then you can install those specific platform lock sections: pipenv sync --categories windows
@matteius for that example yes, it just points out that each platform could have a totally unique dependency, like mac = 1.0, windows = 2.0, linux=3.0. You can't always simplify :) So the poetry way of maintaining all the requirements and re-processing them during sync is more flexible
@thehesiod The problem is you cannot create the pipenv locks on the system you are not on for transitive dependencies or sdists that need to be built on that specific platform, so maintaining all of the requirements in one section is less practical to how pipenv utilizes the pip resolver.
@matteius ya, it would be nice to have the lockfiles be per arch, perhaps like Pipfile.w32.lock or something. Given you can have runtime dynamic dependencies it makes more sense to have one lockfile per arch, perhaps like Pipfile.darwin_x86_64.lock or something. Actually I just found a counterpoint with poetry, it's unable to lock if not all platforms have valid scenarios, like we don't do win32, but because of this line it fails: https://github.com/PyCQA/pylint/blob/v2.15.2/pyproject.toml#L45.
I've tried to find a way to detect if this could be an issue in my Pipfile.lock
. I think the following command detects this. Wouldn't it?
pipenv run grep "Root-Is-Purelib: false" $(dirname $(dirname $(which python3)))/lib/python3*/site-packages * -R --include=WHEEL