pulp
When uploading an invalid rpm such as a non utf8 RPM, users should get a friendly error message
Author: kersom (kersom)
Redmine Issue: 4215, https://pulp.plan.io/issues/4215
When users upload an invalid RPM such as one that has non-utf8 characters in their headers, they should get a 4xx response and a user friendly error message. Currently they receive a 500.
Steps to reproduce.
http POST http://localhost:8000/pulp/api/v3/repositories/ name=foo
export REPO_HREF=$(http :8000/pulp/api/v3/repositories/ | jq -r '.results[] | select(.name == "foo") | ._href')
wget https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-non-utf-8/rpm-with-non-utf-8-1-1.fc25.noarch.rpm
http --form POST http://localhost:8000/pulp/api/v3/artifacts/ file@./rpm-with-non-utf-8-1-1.fc25.noarch.rpm
http POST http://localhost:8000/pulp/api/v3/content/rpm/packages/ relative_path=rpm-with-non-utf-8-1-1.fc25.noarch.rpm artifact="/pulp/api/v3/artifacts/1/" filename=rpm-with-non-utf8.rpm
Traceback
HTTP/1.1 500 Internal Server Error
Content-Length: 20215
Content-Type: text/plain; charset=utf-8
Date: Thu, 29 Nov 2018 15:41:48 GMT
Server: WSGIServer/0.2 CPython/3.7.1
Vary: Cookie
X-Frame-Options: SAMEORIGIN
UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte
Request Method: POST
Request URL: http://localhost:8000/pulp/api/v3/content/rpm/packages/
Django Version: 2.1.3
Python Executable: /home/vagrant/.virtualenvs/pulp/bin/python3
Python Version: 3.7.1
Python Path: ['/home/vagrant/.virtualenvs/pulp/bin', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages', '/home/vagrant/devel/pulp/pulpcore', '/home/vagrant/devel/pulp/plugin', '/home/vagrant/pulp_rpm', '/home/vagrant/.virtualenvs/pulp/lib/python3.7/site-packages']
Server time: Thu, 29 Nov 2018 15:41:48 +0000
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_filters',
'drf_yasg',
'rest_framework',
'pulpcore.app',
'pulp_file.app.PulpFilePluginAppConfig',
'pulp_rpm.app.PulpRpmPluginAppConfig',
'crispy_forms',
'django_extensions']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/exception.py" in inner
34. response = get_response(request)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
126. response = self.process_exception_by_middleware(e, request)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
124. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
54. return view_func(*args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/viewsets.py" in view
116. return self.dispatch(request, *args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
495. response = self.handle_exception(exc)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in handle_exception
455. self.raise_uncaught_exception(exc)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
492. response = handler(request, *args, **kwargs)
File "/usr/lib64/python3.7/contextlib.py" in inner
74. return func(*args, **kwds)
File "/home/vagrant/pulp_rpm/pulp_rpm/app/viewsets.py" in create
96. package = Package.createrepo_to_dict(cr_pkginfo)
File "/home/vagrant/pulp_rpm/pulp_rpm/app/models.py" in createrepo_to_dict
243. 'description': getattr(package, CREATEREPO_PACKAGE_ATTRS.DESCRIPTION) or '',
Exception Type: UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
Exception Value: 'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte
Request information:
USER: admin
GET: No GET data
POST: No POST data
FILES: No FILES data
COOKIES: No cookie data
META:
CONTENT_LENGTH = '137'
CONTENT_TYPE = 'application/json'
[vagrant@p3 ~]$ cat /etc/redhat-release
Fedora release 29 (Twenty Nine)
From: @bmbouter (bmbouter) Date: 2018-11-30T15:54:54Z
The Pulp2 behavior for a RPM to Pulp2 with non utf-8 chars in certain fields would have those invalid chars removed. That's good for being able to sync repositories with invalidly formed RPMs. It's not good because Pulp is mutating the data in unexpected ways. For example now the checksum of that field won't validate correctly after passing through Pulp which is a problem. The real fix is for those RPMs to have their metadata correctly utf-8 encoded.
Note that per the spec in section 2.3.1, RPMs expect utf-8 encoding for various fields, so Pulp3 is correct in rejecting an encoding it can't interpret. Since we aren't getting active bug reports from users, I think we should close this for now. I'm leaving open to go back through triage.
Other ideas and thoughts/concerns are welcome.
From: daviddavis (daviddavis) Date: 2018-12-04T15:51:32Z
Updating issue to better handle invalid RPMs.
