copr
copr copied to clipboard
fedora-copr
Copr tracebacks (httpd 504) on requests to remove large projects
Request: POST https://copr.fedorainfracloud.org/backend/update/
User: None
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/copr/coprs_frontend/coprs/views/misc.py", line 231, in decorated_function
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/share/copr/coprs_frontend/coprs/views/backend_ns/backend_general.py", line 398, in update
request_data = flask.request.json
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 540, in json
return self.get_json()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 575, in get_json
data = self.get_data(cache=cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 405, in get_data
rv = self.stream.read()
^^^^^^^^^^^^^^^^^^
OSError: Apache/mod_wsgi request data read error: Partial results are valid but processing is incomplete.
Request: POST https://copr.fedorainfracloud.org/backend/update/
User: None
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/copr/coprs_frontend/coprs/views/misc.py", line 231, in decorated_function
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/share/copr/coprs_frontend/coprs/views/backend_ns/backend_general.py", line 398, in update
request_data = flask.request.json
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 540, in json
return self.get_json()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 575, in get_json
data = self.get_data(cache=cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 405, in get_data
rv = self.stream.read()
^^^^^^^^^^^^^^^^^^
OSError: Apache/mod_wsgi request data read error: Partial results are valid but processing is incomplete.
See seem to delete builds one-by-one?
OSError: Apache/mod_wsgi request data read error: Partial results are valid but processing is incomplete.
2023-09-26 16:03:27,117 [ERROR][/usr/share/copr/coprs_frontend/coprs/error_handlers.py:42|error_handlers:handle_error][backend: 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83] Response error: 500 Request wasn't successful, there is probably a bug in the Copr code.
2023-09-26 16:03:27,118 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950497
2023-09-26 16:03:27,135 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4963772
2023-09-26 16:03:27,137 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950496
2023-09-26 16:03:27,149 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950495
2023-09-26 16:03:27,158 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950494
2023-09-26 16:03:27,168 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950493
2023-09-26 16:03:27,183 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950492
2023-09-26 16:03:27,185 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950491
2023-09-26 16:03:27,191 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950490
2023-09-26 16:03:27,232 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4963771
2023-09-26 16:03:27,234 [INFO][/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py:1205|builds_logic:delete_build][churchyard] User 'churchyard' removing the build 4950489
[26/Sep/2023:15:59:45 +0000] "POST /coprs/g/python/python3.8/delete/ HTTP/1.1" 504 247 "https://copr.fedorainfracloud.org/coprs/g/python/python3.8/delete/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"
[26/Sep/2023:15:59:57 +0000] "POST /coprs/g/python/python3.9/delete/ HTTP/1.1" 504 247 "https://copr.fedorainfracloud.org/coprs/g/python/python3.9/delete/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"
[26/Sep/2023:16:00:11 +0000] "POST /coprs/g/python/python3.10/delete/ HTTP/1.1" 504 247 "https://copr.fedorainfracloud.org/coprs/g/python/python3.10/delete/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"
[26/Sep/2023:16:00:17 +0000] "POST /coprs/g/python/python3.11/delete/ HTTP/1.1" 504 247 "https://copr.fedorainfracloud.org/coprs/g/python/python3.11/delete/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"
[26/Sep/2023:16:01:40 +0000] "POST /coprs/g/python/python3.8/delete/ HTTP/1.1" 504 247 "https://copr.fedorainfracloud.org/coprs/g/python/python3.8/delete/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"
[26/Sep/2023:16:02:18 +0000] "POST /coprs/g/python/python3.9/delete/ HTTP/1.1" 504 247 "https://copr.fedorainfracloud.org/coprs/g/python/python3.9/delete/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"
Triage time: Ok, so while we are generating just a single action for backend-action processor, we remove the builds from Frontend's database one-by-one, and this likely takes more than 30s for large projects so the request timeouts.
The workaround we used for Python projects was to set --delete-after-days 0.