flit
flit copied to clipboard
pypa
Bad Request for url (400 Client Error) on Gitlab CI
Hi!
Describe the bug:
I am publishing my package in Gitlab CI using flit publish and I am getting these errors when uploading the package to the registry
To Reproduce:
on my local machine works fine when building the package and submitting it to Gitlab registry
Details
(.venv) [brunaldo@fedora oaspy] $ flit publish --repository gitlab
Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno
Found 19 files tracked in git I-flit.sdist
Built sdist: dist/oaspy-2023.8.25.tar.gz I-flit_core.sdist
Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno
Copying package file(s) from /tmp/tmp1warla15/oaspy-2023.8.25/src/oaspy I-flit_core.wheel
Writing metadata files I-flit_core.wheel
Writing the record of files I-flit_core.wheel
Built wheel: dist/oaspy-2023.8.25-py3-none-any.whl I-flit_core.wheel
Using repository at https://gitlab.com/api/v4/projects/47862338/packages/pypi I-flit.upload
Uploading dist/oaspy-2023.8.25-py3-none-any.whl... I-flit.upload
Package is at https://gitlab.com/api/v4/projects/47862338/packages/pypi/oaspy I-flit.upload
Using repository at https://gitlab.com/api/v4/projects/47862338/packages/pypi I-flit.upload
Uploading dist/oaspy-2023.8.25.tar.gz... I-flit.upload
Package is at https://gitlab.com/api/v4/projects/47862338/packages/pypi/oaspy I-flit.upload
...but when I run the pipeline in Gitlab it generates the following error:
Details
$ flit build --no-use-vcs
Fetching list of valid trove classifiers I-flit.validate
Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno
Built sdist: dist/oaspy-2023.8.25.tar.gz I-flit_core.sdist
Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno
Copying package file(s) from /tmp/tmptdjz0g0s/oaspy-2023.8.25/src/oaspy I-flit_core.wheel
Writing metadata files I-flit_core.wheel
Writing the record of files I-flit_core.wheel
Built wheel: dist/oaspy-2023.8.25-py3-none-any.whl I-flit_core.wheel
$ flit publish --no-use-vcs
Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno
Built sdist: dist/oaspy-2023.8.25.tar.gz I-flit_core.sdist
Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno
Copying package file(s) from /tmp/tmpl4fwszuw/oaspy-2023.8.25/src/oaspy I-flit_core.wheel
Writing metadata files I-flit_core.wheel
Writing the record of files I-flit_core.wheel
Built wheel: dist/oaspy-2023.8.25-py3-none-any.whl I-flit_core.wheel
Using repository at https://gitlab.com/api/v4/projects/47862338/packages/pypi I-flit.upload
Uploading dist/oaspy-2023.8.25-py3-none-any.whl... I-flit.upload
Traceback (most recent call last):
File "/usr/local/bin/flit", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/site-packages/flit/__init__.py", line 200, in main
main(args.ini_file, repository, args.pypirc, formats=set(args.format or []),
File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 277, in main
do_upload(built.wheel.file, built.wheel.builder.metadata, pypirc_path, repo_name)
File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 252, in do_upload
upload_file(file, metadata, repo)
File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 245, in upload_file
resp.raise_for_status()
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://gitlab.com/api/v4/projects/47862338/packages/pypi
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
the complete pipeline is here...
Expected behavior:
be able to generate the package and send it to Gitlab Registry (initially) and then to PyPI. 😄
Desktop (please complete the following information):
- OS: Fedora Workstation (38)
- Python version: 3.10.12
- Flit version: 3.9.0
Additional context:
my .gitlab-ci.yml file
image: python:3.10-buster
stages:
- deploy
before_script:
- pip install flit
- flit build --no-use-vcs
deploy_staging:
stage: deploy
variables:
FLIT_USERNAME: $STAGING_PAT_NAME
FLIT_PASSWORD: $STAGING_PAT_TOKEN
FLIT_INDEX_URL: https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi
script:
- flit publish --no-use-vcs
only:
- master
cache:
paths:
- dist/
in the before_script step I am using it that way because it generated another error:
Details
$ flit build
Fetching list of valid trove classifiers I-flit.validate
Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno
Traceback (most recent call last):
File "/usr/local/bin/flit", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/site-packages/flit/__init__.py", line 191, in main
main(args.ini_file, formats=set(args.format or []),
File "/usr/local/lib/python3.10/site-packages/flit/build.py", line 46, in main
sdist_file = sb.build(dist_dir, gen_setup_py=gen_setup_py)
File "/usr/local/lib/python3.10/site-packages/flit_core/sdist.py", line 175, in build
files_to_add = self.apply_includes_excludes(self.select_files())
File "/usr/local/lib/python3.10/site-packages/flit/sdist.py", line 173, in select_files
untracked_deleted = vcs_mod.list_untracked_deleted_files(self.cfgdir)
File "/usr/local/lib/python3.10/site-packages/flit/vcs/git.py", line 12, in list_untracked_deleted_files
outb = check_output(['git', 'ls-files', '--deleted', '--others',
File "/usr/local/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/local/lib/python3.10/subprocess.py", line 503, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/lib/python3.10/subprocess.py", line 971, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'git'
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1
more info about this error, here
some links of interest that I have reviewed:
- using gitlab pipelines to deploy python packages
- flit/issues/364
- gitlab pypi_repository
- flit/pull/416
- gitlab/-/issues/333964
- flit.pypa.io using environment variables
thank you very much! sorry for the bad translation, I don't speak English.
I haven't seen that error before, and I haven't used gitlab registry, but my first thought is that it might be related to authentication. Take a look at https://flit.pypa.io/en/stable/upload.html and check how it's set up to authenticate the upload - maybe this is different between the gitlab pipeline and running locally.
On Sat, 26 Aug 2023, 08:07 Jorge Brunal, @.***> wrote:
Hi!
Describe the bug:
I am publishing my package https://gitlab.com/HomeInside/oaspy in Gitlab CI using flit publish and I am getting these errors when uploading the package to the registry
To Reproduce:
on my local machine works fine when building the package and submitting it to Gitlab registry Details
(.venv) @.*** oaspy] $ flit publish --repository gitlab Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno Found 19 files tracked in git I-flit.sdist Built sdist: dist/oaspy-2023.8.25.tar.gz I-flit_core.sdist Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno Copying package file(s) from /tmp/tmp1warla15/oaspy-2023.8.25/src/oaspy I-flit_core.wheel Writing metadata files I-flit_core.wheel Writing the record of files I-flit_core.wheel Built wheel: dist/oaspy-2023.8.25-py3-none-any.whl I-flit_core.wheel Using repository at https://gitlab.com/api/v4/projects/47862338/packages/pypi I-flit.upload Uploading dist/oaspy-2023.8.25-py3-none-any.whl... I-flit.upload Package is at https://gitlab.com/api/v4/projects/47862338/packages/pypi/oaspy I-flit.upload Using repository at https://gitlab.com/api/v4/projects/47862338/packages/pypi I-flit.upload Uploading dist/oaspy-2023.8.25.tar.gz... I-flit.upload Package is at https://gitlab.com/api/v4/projects/47862338/packages/pypi/oaspy I-flit.upload
...but when I run the pipeline in Gitlab it generates the following error: Details
$ flit build --no-use-vcs Fetching list of valid trove classifiers I-flit.validate Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno Built sdist: dist/oaspy-2023.8.25.tar.gz I-flit_core.sdist Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno Copying package file(s) from /tmp/tmptdjz0g0s/oaspy-2023.8.25/src/oaspy I-flit_core.wheel Writing metadata files I-flit_core.wheel Writing the record of files I-flit_core.wheel Built wheel: dist/oaspy-2023.8.25-py3-none-any.whl I-flit_core.wheel $ flit publish --no-use-vcs Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno Built sdist: dist/oaspy-2023.8.25.tar.gz I-flit_core.sdist Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno Copying package file(s) from /tmp/tmpl4fwszuw/oaspy-2023.8.25/src/oaspy I-flit_core.wheel Writing metadata files I-flit_core.wheel Writing the record of files I-flit_core.wheel Built wheel: dist/oaspy-2023.8.25-py3-none-any.whl I-flit_core.wheel Using repository at https://gitlab.com/api/v4/projects/47862338/packages/pypi I-flit.upload Uploading dist/oaspy-2023.8.25-py3-none-any.whl... I-flit.upload Traceback (most recent call last): File "/usr/local/bin/flit", line 8, in
sys.exit(main()) File "/usr/local/lib/python3.10/site-packages/flit/init.py", line 200, in main main(args.ini_file, repository, args.pypirc, formats=set(args.format or []), File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 277, in main do_upload(built.wheel.file, built.wheel.builder.metadata, pypirc_path, repo_name) File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 252, in do_upload upload_file(file, metadata, repo) File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 245, in upload_file resp.raise_for_status() File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://gitlab.com/api/v4/projects/47862338/packages/pypi Cleaning up project directory and file based variables 00:01 ERROR: Job failed: exit code 1 the complete pipeline is here... https://gitlab.com/HomeInside/oaspy/-/jobs/4958446556
Expected behavior:
be able to generate the package and send it to Gitlab Registry (initially) and then to PyPI. 😄
Desktop (please complete the following information):
- OS: Fedora Workstation (38)
- Python version: 3.10.12
- Flit version: 3.9.0
Additional context:
my .gitlab-ci.yml file
image: python:3.10-buster stages:
- deploy before_script:
- pip install flit
- flit build --no-use-vcs deploy_staging: stage: deploy variables: FLIT_USERNAME: $STAGING_PAT_NAME FLIT_PASSWORD: $STAGING_PAT_TOKEN FLIT_INDEX_URL: https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi script: - flit publish --no-use-vcs only: - master cache: paths:
- dist/
in the before_script step I am using it that way because it generated another error: Details
$ flit build Fetching list of valid trove classifiers I-flit.validate Version number normalised: '2023.08.25' -> '2023.8.25' (see PEP 440) W-flit_core.versionno Traceback (most recent call last): File "/usr/local/bin/flit", line 8, in
sys.exit(main()) File "/usr/local/lib/python3.10/site-packages/flit/init.py", line 191, in main main(args.ini_file, formats=set(args.format or []), File "/usr/local/lib/python3.10/site-packages/flit/build.py", line 46, in main sdist_file = sb.build(dist_dir, gen_setup_py=gen_setup_py) File "/usr/local/lib/python3.10/site-packages/flit_core/sdist.py", line 175, in build files_to_add = self.apply_includes_excludes(self.select_files()) File "/usr/local/lib/python3.10/site-packages/flit/sdist.py", line 173, in select_files untracked_deleted = vcs_mod.list_untracked_deleted_files(self.cfgdir) File "/usr/local/lib/python3.10/site-packages/flit/vcs/git.py", line 12, in list_untracked_deleted_files outb = check_output(['git', 'ls-files', '--deleted', '--others', File "/usr/local/lib/python3.10/subprocess.py", line 421, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/local/lib/python3.10/subprocess.py", line 503, in run with Popen(*popenargs, **kwargs) as process: File "/usr/local/lib/python3.10/subprocess.py", line 971, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.10/subprocess.py", line 1863, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'git' Cleaning up project directory and file based variables 00:00 ERROR: Job failed: exit code 1 more info about this error, here https://gitlab.com/HomeInside/oaspy/-/jobs/4958423656
some links of interest that I have reviewed:
- using gitlab pipelines to deploy python packages @.***/using-gitlab-pipelines-to-deploy-python-packages-in-production-and-staging-environments-8ab7dc979274>
- flit/issues/364 https://github.com/pypa/flit/issues/364
- gitlab pypi_repository https://docs.gitlab.com/ee/user/packages/pypi_repository/
- flit/pull/416 https://github.com/pypa/flit/pull/416
- gitlab/-/issues/333964 https://gitlab.com/gitlab-org/gitlab/-/issues/333964
- flit.pypa.io using environment variables https://flit.pypa.io/en/stable/upload.html#using-environment-variables
thank you very much! sorry for the bad translation, I don't speak English.
— Reply to this email directly, view it on GitHub https://github.com/pypa/flit/issues/654, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACQB5NIGQ5J777RXVG6MJLXXGHB7ANCNFSM6AAAAAA37KP4NM . You are receiving this because you are subscribed to this thread.Message ID: @.***>
@takluyver thanks for the reply, I have been doing some research about what you said with the following results:
on my local machine doing a deployment to PyPI.
Details
(.venv) [brunaldo@fedora oaspy] $ flit publish
Version number normalised: '2023.08.30' -> '2023.8.30' (see PEP 440) W-flit_core.versionno
Found 20 files tracked in git I-flit.sdist
Built sdist: dist/oaspy-2023.8.30.tar.gz I-flit_core.sdist
Version number normalised: '2023.08.30' -> '2023.8.30' (see PEP 440) W-flit_core.versionno
Copying package file(s) from /tmp/tmp01sjmf0g/oaspy-2023.8.30/src/oaspy I-flit_core.wheel
Writing metadata files I-flit_core.wheel
Writing the record of files I-flit_core.wheel
Built wheel: dist/oaspy-2023.8.30-py3-none-any.whl I-flit_core.wheel
Using repository at https://upload.pypi.org/legacy/ I-flit.upload
Uploading dist/oaspy-2023.8.30-py3-none-any.whl... I-flit.upload
Package is at https://pypi.org/project/oaspy/ I-flit.upload
Using repository at https://upload.pypi.org/legacy/ I-flit.upload
Uploading dist/oaspy-2023.8.30.tar.gz... I-flit.upload
Package is at https://pypi.org/project/oaspy/ I-flit.upload
everything working as I should expect. :ok_hand:
since doing the release from GitLab, I get a similar issue:
Details
$ flit publish
Version number normalised: '2023.08.30' -> '2023.8.30' (see PEP 440) W-flit_core.versionno
Found 20 files tracked in git I-flit.sdist
Built sdist: dist/oaspy-2023.8.30.tar.gz I-flit_core.sdist
Version number normalised: '2023.08.30' -> '2023.8.30' (see PEP 440) W-flit_core.versionno
Copying package file(s) from /tmp/tmpqq_k1skd/oaspy-2023.8.30/src/oaspy I-flit_core.wheel
Writing metadata files I-flit_core.wheel
Writing the record of files I-flit_core.wheel
Built wheel: dist/oaspy-2023.8.30-py3-none-any.whl I-flit_core.wheel
Using repository at https://upload.pypi.org/legacy/ I-flit.upload
Uploading dist/oaspy-2023.8.30-py3-none-any.whl... I-flit.upload
Traceback (most recent call last):
File "/usr/local/bin/flit", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/site-packages/flit/__init__.py", line 200, in main
main(args.ini_file, repository, args.pypirc, formats=set(args.format or []),
File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 277, in main
do_upload(built.wheel.file, built.wheel.builder.metadata, pypirc_path, repo_name)
File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 252, in do_upload
upload_file(file, metadata, repo)
File "/usr/local/lib/python3.10/site-packages/flit/upload.py", line 245, in upload_file
resp.raise_for_status()
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for more information. for url: https://upload.pypi.org/legacy/
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
the complete pipeline is here...
I'm using environment variables... in the configuration section, but not a .pypirc file in gitlab.
You can specify a server to upload to with FLIT_INDEX_URL, and pass credentials with FLIT_USERNAME and FLIT_PASSWORD. Environment variables take precedence over the config file, except if you use the --repository option to explicitly pick a server from the config file.
my .gitlab-ci.yml file
image: python:3.10-buster
stages:
- deploy
before_script:
- pip install flit
- flit build --no-use-vcs
# is commented due to some bugs, for now
# deploy_staging:
# stage: deploy
# variables:
# FLIT_USERNAME: $STAGING_PAT_NAME
# FLIT_PASSWORD: $STAGING_PAT_TOKEN
# # FLIT_INDEX_URL: https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi
# script:
# - flit publish --no-use-vcs
# only:
# - master
# cache:
# paths:
# - dist/
deploy_production:
stage: deploy
variables:
FLIT_USERNAME: __token__
FLIT_PASSWORD: $PRODUCTION_PYPI_TOKEN
script:
- flit publish
only:
- master
cache:
paths:
- dist/
for the time being I will disable CI/CD to deploy PYPI and rather generate artifacts
you can close this issue, but in the (distant) future, will it be possible to reopen it in case a better solution is found?