httpexception rendering crashes if wsgi environ contains non-utf8 bytestrings

[swooster-submitnet]

I run Pyramid with mod_wsgi and hackers probing for vulnerabilities have been able to cause crashes by crafting malicious requests with 0x80 bytes. Here's some minimal code replicating what I'm seeing on production:

from pyramid.config import Configurator
from pyramid.httpexceptions import HTTPSeeOther
from webob import Request

def hello_world(request):
    return HTTPSeeOther("http://example.com/")

config = Configurator()
config.add_route('hello', '/')
config.add_view(hello_world, route_name='hello')
app = config.make_wsgi_app()

request = Request.blank('/', method='\x80')
response = request.get_response(app)

Ideally I'd expect Pyramid to respond with a 405 Method Not Allowed or 400 Bad Request, or maybe even a 403 Forbidden or 404 Not Found. Instead it raises a UnicodeDecodeError:

  File "./crash.py", line 16, in <module>
    response = request.get_response(app)
  File "/usr/local/lib/python2.7/site-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/site-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/pyramid/router.py", line 243, in __call__
    return response(request.environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/pyramid/httpexceptions.py", line 286, in __call__
    self.prepare(environ)
  File "/usr/local/lib/python2.7/site-packages/pyramid/httpexceptions.py", line 261, in prepare
    args[k] = escape(v)
  File "/usr/local/lib/python2.7/site-packages/pyramid/httpexceptions.py", line 147, in _no_escape
    value = text_(value, 'utf-8')
  File "/usr/local/lib/python2.7/site-packages/pyramid/compat.py", line 45, in text_
    return s.decode(encoding, errors)
  File "/usr/local/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 0: invalid start byte

The problem seems to be that when used as a WSGI handler, certain pyramid.httpexceptions.HTTPExceptions attempt to decode bytes in the WSGI environ into strings via UTF-8 (for use as template arguments), but don't handle decoding failures.

[mcdonc]

I don't think "crashes" is the right word to use for what happens, is it? An exception is logged, but the server stays running, right?

[swooster-submitnet]

You're right. I tend to call 500 errors due to unhandled exceptions "crashes", but perhaps another word would be better.

[digitalresistor]

Just linking this: Pylons/webob#161

[gnmerritt]

I know this is an old issue, but it's resolvable (no more log spam) with something like the following:

from pyramid.exceptions import URLDecodeError
from pyramid.view import exception_view_config

@exception_view_config(context=URLDecodeError, renderer='json')
def urldecode_json_handler(context, request):
    request.response.status = 404
    return {'status': 404}

after which things like curl localhost:6543/%c0%2e%c0%2e\x5C%c0%2e%c0%2e\x5C%c0%2e%c0%2e\x5Cboot.ini won't 500