[BUG] beaker-proxy hangs on a GET request resulting in all Beaker jobs hanging - server/controller reboot necessary

[desnesn]

DESCRIPTION
The systems on my company are scanned for security vulnerabilities and the beaker-proxy service is hanging with one GET request made by the vulnerability tool.

After the GET request searching for an specific vulnerability is received, all beaker jobs get stuck in a Waiting or Updating stage and don't move forward. Restarting beaker-proxy and beaker-watchdog services does not get Beaker unstuck - only thing that works is literally restarting everything on the Beaker Controllers and Beaker Server!

Stack trace on the beaker-proxy.log suggests the failure is on the UTF-8 decodification of this GET request that contains a 0x80 byte

...
Jun 20 07:45:13 <hostname> beaker-proxy: ::ffff:<IP OF VULN. SCANNER> - - [2022-06-20 07:45:13] "GET .\.\.\.\.\.\.\.\.\.\/winnt/win.ini HTTP/1.1" 404 342 0.000217
Jun 20 07:45:13 <hostname> beaker-proxy: ::ffff:<IP OF VULN. SCANNER> - - [2022-06-20 07:45:13] "GET /nessus\..\..\..\..\..\..\windows\win.ini HTTP/1.1" 404 342 0.000257
Jun 20 07:45:14 <hostname> beaker-proxy: ::ffff:<IP OF VULN. SCANNER> - - [2022-06-20 07:45:14] "GET /nessus\..\..\..\..\..\..\winnt\win.ini HTTP/1.1" 404 342 0.000303
Jun 20 07:45:14 <hostname> beaker-proxy[12587]: bkr.labcontroller.main ERROR Error handling request GET /<80>../<80>../<80>../<80>../<80>../<80>../windows/win.ini
Jun 20 07:45:14 <hostname> beaker-proxy[12587]: Traceback (most recent call last):
Jun 20 07:45:14 <hostname> beaker-proxy: Traceback (most recent call last):
Jun 20 07:45:14 <hostname> beaker-proxy: File "/usr/lib64/python2.7/site-packages/gevent/pywsgi.py", line 884, in handle_one_response
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:   File "/usr/lib/python2.7/site-packages/bkr/labcontroller/main.py", line 169, in _log_failed_requests
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:     return func(environ, start_response)
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:   File "/usr/lib/python2.7/site-packages/werkzeug/wrappers.py", line 285, in application
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:     return f(*args[:-2] + (request,))(*args[-2:])
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:   File "/usr/lib/python2.7/site-packages/bkr/labcontroller/main.py", line 133, in __call__
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:     if req.path in ('/', '/RPC2', '/server'):
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:   File "/usr/lib/python2.7/site-packages/werkzeug/utils.py", line 71, in __get__
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:     value = self.func(obj)
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:   File "/usr/lib/python2.7/site-packages/werkzeug/wrappers.py", line 503, in path
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:     self.charset, self.encoding_errors)
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:   File "/usr/lib/python2.7/site-packages/werkzeug/_compat.py", line 92, in wsgi_decoding_dance
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:     return s.decode(charset)
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:   File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
Jun 20 07:45:14 <hostname> beaker-proxy[12587]:     return codecs.utf_8_decode(input, errors, True)
Jun 20 07:45:14 <hostname> beaker-proxy[12587]: UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 1: invalid start byte
...

VERSION-RELEASE-NUMBER
[root@ beaker]# rpm -qa | grep beaker-lab beaker-lab-controller-28.2-1.el7.noarch

REPRODUCE
Still trying to figure out a python or shell script to reproduce the stack trace. Trying something like this:

#!/usr/bin/python3
# -*- coding: utf-8 -*-

import codecs
import requests

url = "https://<fqdn>/\x81../\x80../\x80../\x80../\x80../\x80../windows/win.ini HTTP/1.1"

x = requests.get( codecs.encode(url, encoding='utf-8', errors='strict') )

print(x.status_code)

print(x.text)

But still haven't quite been able to reproduce it.

ACTUAL BEHAVIOR
The beaker-proxy service hangs, and thus all beaker jobs hang - reboot of everything is necessary.

EXPECTED BEHAVIOR
No hang on anything - just a 404 error should be returned with no stack trace on the logs.

ADITIONAL CONTEXT
Code that fails:

[root@<fqdn> ~]# rpm -qf /usr/lib64/python2.7/encodings/utf_8.py
python-libs-2.7.5-90.el7.x86_64

[root@<fqdn> beaker]# cat /usr/lib64/python2.7/encodings/utf_8.py
""" Python 'utf-8' Codec


Written by Marc-Andre Lemburg ([email protected]).

(c) Copyright CNRI, All Rights Reserved. NO WARRANTY.

"""
import codecs

### Codec APIs

encode = codecs.utf_8_encode

def decode(input, errors='strict'):
    return codecs.utf_8_decode(input, errors, True)
...

Maybe the GET request need to be further sanitized somewhere prior to trying to decode it?

Thanks in advance for any help on this issue :-)

[desnesn]

Somewhere on python-werkzeug? This thread caught the eye: https://github.com/pallets/werkzeug/issues/808

And this:

[root@<fqdn> beaker]$ rpm -qa | grep werkzeug
python-werkzeug-0.9.1-2.el7.noarch

What is the current recommended version of python-werkzeug to be run on Beaker?

[desnesn]

This is also a clue?

[root@<fqdn> ~]# cat beaker.bug.py 
#!/usr/bin/python3

s = b'\x80\xf8\xe7'
print(s.decode('latin-1'))
print(s.decode('windows-1252'))
print(s.decode('UTF-8'))

[root@<fqdn> ~]# ./beaker.bug.py 
øç
€øç
Traceback (most recent call last):
  File "./beaker.bug.3.py", line 6, in <module>
    print(s.decode('UTF-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte