EasyNMT icon indicating copy to clipboard operation
EasyNMT copied to clipboard

Maybe http3 related bug or not?

Open TheMY3 opened this issue 2 years ago • 1 comments

Try to use mbart50_m2m model and when I pass 1 000+ chars, sometimes it can be 1000, sometimes 3000, I got 504 error from my CF and next trace in logs:

[2022-06-08 11:22:59 +0000] [21] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/h11/_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 372, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/main.py", line 105, in translate_post
    return await translate(**data)
  File "/app/main.py", line 55, in translate
    x = await async_client.post(BACKEND_URL+'/translate', json=data, timeout=3600)
  File "/usr/local/lib/python3.8/site-packages/http3/client.py", line 392, in post
    return await self.request(
  File "/usr/local/lib/python3.8/site-packages/http3/client.py", line 541, in request
    response = await self.send(
  File "/usr/local/lib/python3.8/site-packages/http3/client.py", line 140, in send
    response = await self.send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/http3/client.py", line 177, in send_handling_redirects
    response = await self.dispatch.send(
  File "/usr/local/lib/python3.8/site-packages/http3/dispatch/connection_pool.py", line 130, in send
    raise exc
  File "/usr/local/lib/python3.8/site-packages/http3/dispatch/connection_pool.py", line 120, in send
    response = await connection.send(
  File "/usr/local/lib/python3.8/site-packages/http3/dispatch/connection.py", line 59, in send
    response = await self.h11_connection.send(request, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/http3/dispatch/http11.py", line 58, in send
    http_version, status_code, headers = await self._receive_response(timeout)
  File "/usr/local/lib/python3.8/site-packages/http3/dispatch/http11.py", line 130, in _receive_response
    event = await self._receive_event(timeout)
  File "/usr/local/lib/python3.8/site-packages/http3/dispatch/http11.py", line 161, in _receive_event
    event = self.h11_state.next_event()
  File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 439, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/usr/local/lib/python3.8/site-packages/h11/_util.py", line 72, in _reraise_as_remote_protocol_error
    raise self
  File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 422, in next_event
    self._process_event(self.their_role, event)
  File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 238, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "/usr/local/lib/python3.8/site-packages/h11/_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "/usr/local/lib/python3.8/site-packages/h11/_state.py", line 251, in _fire_event_triggered_transitions
    raise LocalProtocolError(
h11._util.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
[2022-06-08 11:22:59 +0000] [16] [WARNING] Worker with pid 160 was terminated due to signal 9
[2022-06-08 11:22:59 +0000] [962] [INFO] Booting worker with pid: 962
[2022-06-08 11:23:19 +0000] [962] [INFO] Started server process [962]
[2022-06-08 11:23:19 +0000] [962] [INFO] Waiting for application startup.
[2022-06-08 11:23:19 +0000] [962] [INFO] Application startup complete.

I found similar PR in another library - https://github.com/encode/httpx/issues/1478 and found that you also use http3 in requirements. Can you help with this?

TheMY3 avatar Jun 08 '22 11:06 TheMY3

One more update - its happens when request executed more than one minute, is it limit for request in 1 minute? Default timeout should be 120 sec (https://github.com/UKPLab/EasyNMT/tree/main/docker#environment-variables)

TheMY3 avatar Jun 08 '22 12:06 TheMY3