Cytomine-core
Cytomine-core copied to clipboard
cytomine
Problem uploading annotation: 413 Request Entity Too Large
Hello,
I have been working on a project with cytomine and there is a problem when trying to upload large (~2 million points) annotations.
I am not sure if the problem comes from the python client or from cytomine core. This may not even be an issue, as this is a very large annotation. However I was unable to find the maximum number of points accepted by cytomine in the documentation.
Here is a minimal example to reproduce the issue:
from argparse import ArgumentParser
from cytomine import Cytomine
from cytomine.models import ImageInstance
from cytomine.models import Annotation
from shapely.geometry import Polygon
import numpy as np
def main():
parser = ArgumentParser(prog="Cytomine Python client example")
parser.add_argument('--host', required=True, help="The Cytomine host")
parser.add_argument('--public_key', required=True, help="The Cytomine public key")
parser.add_argument('--private_key', required=True, help="The Cytomine private key")
parser.add_argument('--image_id', required=True, help="The id of the image to annotate")
params = parser.parse_args()
with Cytomine(params.host, params.public_key, params.private_key) as cytomine:
# Fetch image
image= ImageInstance().fetch(id=params.image_id)
# Generic Polygon with 2 million elements
x = np.linspace(start=0,stop=2*np.pi, num=int(2e6))
y = np.sin(x)*500+500
x = np.cos(x)*500+500
xy = np.stack([x,y], axis=1)
# Convert to Polygon
geometry = Polygon(xy)
# Try to upload the annotation
Annotation(location=geometry.wkt, id_image=image.id).save()
if __name__ == "__main__":
main()
And here is the output of the previous code:
[2022-07-25 14:55:31,724][INFO] [GET] /server/ping | 200
[2022-07-25 14:55:31,747][INFO] [GET] [currentuser] CURRENT USER - 14696662 : bdorra | 200
[2022-07-25 14:55:31,769][INFO] [GET] [imageinstance] 13878732 : M92_PAS_2021-05-28.jpg | 200
[2022-07-25 14:55:33,426][ERROR] [POST] [annotation] None | 413 Request Entity Too Large (<html>
<head><title>413 Request Entity Too Large</title></head>
<body bgcolor="white">
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx/1.10.3</center>
</body>
</html>
)
This code works as expected with a smaller number of points (ex:2e3).
Hello !
I encountered this issue again. After closer inspection it seems to be linked with the default limit of the nginx server for client payload size: 1MB.
The length of the polygon representation has to be limited to fit this limit.
The fix is probably to reconfigure the server by setting a higher client_max_body_size value. See this link for more info: https://www.keycdn.com/support/413-request-entity-too-large.
Hello,
Do you succeed to modify the nginx configuration ? It is located in the conf/nginx/nginx.conf file in the Cytomine-bootstrap folder.
We don't exactly have a defined limit in our documentation but heavy upload take a long time to Cytomine to proceed it and it is recommanded to split it in mutiple chunks.
I think @bathienle has done a study about the optimal size of the chunk. @bathienle do you confirm ?
Hello @geektortoise ,
I don't have direct access to that server (the data is on cloud.cytomine), so I cannot modify the config.
Yes splitting into multiple chunks would solve the problem, but since there is just 1 large polygon incremental simplification until it weights less than 1MB may be a better solution (several MBs for a polygon is a bit much anyway).
I was just surprised at first since uploading smaller polygons had never caused a problem.
EDIT: I had closed the issue but it may be worth waiting for more input in case I missed something.
Hello,
Do you succeed to modify the nginx configuration ? It is located in the conf/nginx/nginx.conf file in the Cytomine-bootstrap folder.
We don't exactly have a defined limit in our documentation but heavy upload take a long time to Cytomine to proceed it and it is recommanded to split it in mutiple chunks.
I think @bathienle has done a study about the optimal size of the chunk. @bathienle do you confirm ?
Hello, sorry for the late reply. The optimal size of the chunk is between 10 and 20. However, larger chunk size can be set but it will be slower. The study was about uploading a large number of annotations but not one annotation with a large number of points. So like you suggest, I would simplify the polygon until it is possible to upload it without any issues.
