cvat
cvat copied to clipboard
Published
20 hours ago •
cvat-ai
cvat-ai
Server crashes if image chunk size is too big > 512 Mb
Actions before raising this issue
- [X] I searched the existing issues and did not find anything similar.
- [X] I read/searched the docs
Steps to Reproduce
-
Create a new task with this collection of pcd files:
-
Try to open the job
-
The server returns 500 while trying to load and in there is a crash in the logs
2024-05-28 13:21:42,169 DEBG 'uvicorn-0' stderr output:
[2024-05-28 13:21:42,167] ERROR django.request: Internal Server Error: /api/jobs/588/data
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 822, in send_packed_command
self._sock.sendall(item)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 534, in thread_handler
raise exc_info[1]
File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 42, in inner
response = await get_response(request)
File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
response = await wrapped_callback(
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 479, in __call__
ret: _R = await loop.run_in_executor(
File "/opt/venv/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
return func(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
return view_func(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
return self.dispatch(request, *args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/home/django/cvat/apps/engine/views.py", line 1903, in data
return data_getter(request, db_job.segment.start_frame,
File "/home/django/cvat/apps/engine/views.py", line 727, in __call__
return super().__call__(request, start, stop, db_data)
File "/home/django/cvat/apps/engine/views.py", line 657, in __call__
buff, mime_type = frame_provider.get_chunk(self.number, self.quality)
File "/home/django/cvat/apps/engine/frame_provider.py", line 207, in get_chunk
return self._loaders[quality].get_chunk_path(chunk_number, quality, self._db_data)
File "/home/django/cvat/apps/engine/cache.py", line 80, in get_task_chunk_data_with_mime
item = self._get_or_set_cache_item(
File "/home/django/cvat/apps/engine/cache.py", line 68, in _get_or_set_cache_item
item = create_item()
File "/home/django/cvat/apps/engine/cache.py", line 55, in create_item
self._cache.set(key, item)
File "/opt/venv/lib/python3.10/site-packages/django/core/cache/backends/redis.py", line 191, in set
self._cache.set(key, value, self.get_backend_timeout(timeout))
File "/opt/venv/lib/python3.10/site-packages/django/core/cache/backends/redis.py", line 108, in set
client.set(key, value, ex=timeout)
File "/opt/venv/lib/python3.10/site-packages/redis/commands/core.py", line 2302, in set
return self.execute_command("SET", *pieces, **options)
File "/opt/venv/lib/python3.10/site-packages/redis/client.py", line 1258, in execute_command
return conn.retry.call_with_retry(
File "/opt/venv/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
fail(error)
File "/opt/venv/lib/python3.10/site-packages/redis/client.py", line 1262, in <lambda>
lambda error: self._disconnect_raise(conn, error),
File "/opt/venv/lib/python3.10/site-packages/redis/client.py", line 1248, in _disconnect_raise
raise error
File "/opt/venv/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/opt/venv/lib/python3.10/site-packages/redis/client.py", line 1259, in <lambda>
lambda: self._send_command_parse_response(
File "/opt/venv/lib/python3.10/site-packages/redis/client.py", line 1234, in _send_command_parse_response
conn.send_command(*args)
File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 840, in send_command
self.send_packed_command(
File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 833, in send_packed_command
raise ConnectionError(f"Error {errno} while writing to socket. {errmsg}.")
redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.
2024-05-28 13:21:42,170 DEBG 'uvicorn-0' stdout output:
INFO: 172.17.0.7:0 - "GET /api/jobs/588/data?org=&quality=compressed&type=chunk&number=0 HTTP/1.0" 500 Internal Server Error
Expected Behavior
The job is opening without a crashing server.
Possible Solution
When I create a task by splitting this set of files into several smaller ones, the task loads correctly
Context
No response
Environment
Server version: 2.10.2
Core version: 14.1.0
Canvas version: 2.19.1
UI version: 1.61.3
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
313b516a88ec gcr.io/iguazio/alpine:3.17 "/bin/sh -c '/bin/sl…" 4 hours ago Up 4 hours nuclio-local-storage-reader
6ff6a699543f cvat/server:v2.10.2 "./backend_entrypoin…" 5 weeks ago Up 2 days 8080/tcp cvat_server
bb31577677e0 cvat.onnx.wongkinyiu.yolov7:latest "processor" 3 months ago Up 2 days (healthy) 0.0.0.0:32768->8080/tcp, :::32768->8080/tcp nuclio-nuclio-onnx-wongkinyiu-yolov7
58d8c150e120 cvat/server:v2.10.2 "./backend_entrypoin…" 3 months ago Up 2 days 8080/tcp cvat_worker_annotation
44faad47461b quay.io/nuclio/dashboard:1.11.24-amd64 "/docker-entrypoint.…" 3 months ago Up 2 days (healthy) 80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp nuclio
c92c9dc59126 cvat/ui:v2.10.2 "/docker-entrypoint.…" 3 months ago Up 2 days 80/tcp cvat_ui
08de5d80177a cvat/server:v2.10.2 "./backend_entrypoin…" 3 months ago Up 2 days 8080/tcp cvat_worker_quality_reports
2539f26b757b cvat/server:v2.10.2 "./backend_entrypoin…" 3 months ago Up 2 days 8080/tcp cvat_worker_webhooks
24d164eceea9 cvat/server:v2.10.2 "./backend_entrypoin…" 3 months ago Up 2 days 8080/tcp cvat_worker_import
57a1d9beae70 cvat/server:v2.10.2 "./backend_entrypoin…" 3 months ago Up 2 days 8080/tcp cvat_utils
b719a17c7ab3 cvat/server:v2.10.2 "./backend_entrypoin…" 3 months ago Up 2 days 8080/tcp cvat_worker_analytics_reports
db6e950d799a cvat/server:v2.10.2 "./backend_entrypoin…" 3 months ago Up 2 days 8080/tcp cvat_worker_export
8faca5315bed timberio/vector:0.26.0-alpine "/usr/local/bin/vect…" 3 months ago Up 2 days cvat_vector
9eccec23bd9d traefik:v2.10 "/entrypoint.sh trae…" 3 months ago Up 2 days 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp traefik
88e8c039a55b apache/kvrocks:2.7.0 "kvrocks -c /var/lib…" 3 months ago Up 2 days (healthy) 6666/tcp cvat_redis_ondisk
b7b44c5d9557 redis:7.2.3-alpine "docker-entrypoint.s…" 3 months ago Up 2 days 6379/tcp cvat_redis_inmem
be87ecfe416d clickhouse/clickhouse-server:23.11-alpine "/entrypoint.sh" 3 months ago Up 2 days 8123/tcp, 9000/tcp, 9009/tcp cvat_clickhouse
3e39d6d1da34 postgres:15-alpine "docker-entrypoint.s…" 3 months ago Up 2 days 5432/tcp cvat_db
cdca097c27e8 openpolicyagent/opa:0.45.0-rootless "/opa run --server -…" 3 months ago Up 2 days cvat_opa
# docker --version
Docker version 23.0.1, build a5ee5b1
/ # docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
cvat.pth.dschoerk.transt latest 5ff0598b1ad0 3 months ago 1.42GB
cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco latest 35dbf34336f1 3 months ago 1.94GB
cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco.base latest a361c0d353cf 3 months ago 1.88GB
cvat.openvino.omz.public.faster_rcnn_inception_resnet_v2_atrous_coco latest 0a6bf6da7cd4 3 months ago 1.73GB
cvat.openvino.omz.public.faster_rcnn_inception_resnet_v2_atrous_coco.base latest 4c60dc98baff 3 months ago 1.67GB
cvat.openvino.omz.intel.text-detection-0004 latest cd102ee7c8c6 3 months ago 1.5GB
cvat.openvino.omz.intel.text-detection-0004.base latest 527ab1dfda6e 3 months ago 1.45GB
cvat.openvino.omz.intel.semantic-segmentation-adas-0001 latest 4a67fc92e65f 3 months ago 1.51GB
cvat.openvino.omz.intel.semantic-segmentation-adas-0001.base latest 4b83055c9665 3 months ago 1.46GB
cvat.openvino.omz.intel.person-reidentification-retail-0277 latest 045ada2caecb 3 months ago 1.63GB
cvat.openvino.omz.intel.person-reidentification-retail-0277.base latest 57e5f5b3dbfc 3 months ago 1.57GB
cvat.openvino.omz.intel.face-detection-0205 latest b553f4a167a8 3 months ago 1.51GB
cvat.openvino.omz.intel.face-detection-0205.base latest beedfac5e088 3 months ago 1.46GB
cvat.openvino.dextr latest 93d12e723174 3 months ago 1.68GB
cvat.openvino.dextr.base latest d5eddecadb29 3 months ago 1.63GB
cvat.onnx.wongkinyiu.yolov7 latest e443f3c05b37 3 months ago 770MB
cvat.openvino.base latest f8c819411853 3 months ago 1.43GB
traefik v2.10 ee69e8120b64 4 months ago 153MB
gcr.io/iguazio/alpine 3.17 eaba187917cc 4 months ago 7.06MB
cvat/ui v2.10.2 a83357de1feb 4 months ago 143MB
cvat/server v2.10.2 ee6648bb036d 4 months ago 3.02GB
clickhouse/clickhouse-server 23.11-alpine ddd2efb58fe7 4 months ago 910MB
postgres 15-alpine 478703aef7f8 5 months ago 240MB
apache/kvrocks 2.7.0 373063f3f9d4 5 months ago 37.3MB
redis 7.2.3-alpine d2d4688fcebe 5 months ago 41MB
grafana/grafana-oss 10.1.2 31656ec60d2e 8 months ago 391MB
quay.io/nuclio/handler-builder-python-onbuild 1.11.24-amd64 94caa75b7738 10 months ago 55.9MB
quay.io/nuclio/dashboard 1.11.24-amd64 86a4ab0cb6f4 10 months ago 250MB
cvat/server latest 19ef97c9cc19 17 months ago 4.71GB
timberio/vector 0.26.0-alpine d8ecc9831523 18 months ago 122MB
openpolicyagent/opa 0.45.0-rootless 8723f2dc306a 19 months ago 84.3MB
cvat/ui v2.2.0 822d202cfca2 20 months ago 51.2MB
cvat/server v2.2.0 0dba6fa26ad3 20 months ago 4.63GB
postgres 10 1cad456b3a24 23 months ago 202MB
openvino/cvat_server latest 041f75bb1d7e 2 years ago 5.95GB
cvat_kibana latest 5f2f95ad9ef4 2 years ago 493MB
cvat_logstash latest a8ea37ce806a 2 years ago 674MB
cvat_elasticsearch latest cac2fd48f1aa 2 years ago 678MB
postgres 10-alpine 2c86947136ab 2 years ago 79.9MB
openvino/cvat_ui v1.7.0 2b45ff0ccf48 2 years ago 49MB
openvino/cvat_server v1.7.0 7720e30a355d 2 years ago 4.71GB
openpolicyagent/opa 0.34.2-rootless f85ee8a15a91 2 years ago 71.9MB
traefik v2.4 de1a7c9d5d63 2 years ago 92MB
redis 4.0-alpine e3dd0e49bca5 4 years ago 20.4MB
quay.io/nuclio/uhttpc 0.0.1-amd64 5c59b3d31aa8 6 years ago 3.96MB
