QCFractal Completed tasks not being pushed because server returns Entity Too Large

Describe the bug

Workers have been recently marking jobs stale, consistently (several per day). After tracing it back to interface/client.py#L235 (in FractalClient::_request), the logs show

[W 200515 22:51:33 managers:579] Post complete tasks was not successful. Attempting again on next update. Error:
[W 200515 22:51:33 managers:580] Server communication failure. Reason: Request Entity Too Large
[I 200515 22:51:33 managers:616] Tried to push 2 complete tasks to the server (2 success / 0 fail).

(I locally modified queue/manager.py#L579 to show this the return message from FractalClient)

This will occur until it hits the retries threshold, then bail. In all failures, the same error is given.

To Reproduce

Spin up managers, wait ~12 hours. I can provide task IDs if they are useful for debugging; I can't tell which job is causing the issues, but I can collect all pulled tasks up to that point.

May 16 '20 06:05 trevorgokey

Ok now that makes sense. This error doesn't come from QCFractal, but from nginx (and now I see it in the nginx error log). For a short term fix, I can increase that limit. Longer term, it might be good to chunk the uploads: https://requests.readthedocs.io/en/master/user/advanced/#chunk-encoded-requests

These seem to be some pretty large results. We currently have a limit of 50M but that is easily changed. Can you give me some of the task ids? I am just curious as to why they are so large

May 16 '20 12:05 bennybp

Sure, a quick grep for duplicate ids that appear before a stale update since May 11 is below. In particular, 10499131 might be of interest

standard/log:    {'id': '10411286'
standard/log:    {'id': '10411469'
standard/log:    {'id': '10430259'
standard/log:    {'id': '10430426'
standard/log:    {'id': '10430617'
standard/log:    {'id': '10430707'
standard/log:    {'id': '10430760'
standard/log:    {'id': '10431318'
standard/log:    {'id': '10431326'
standard/log:    {'id': '10431549'
standard/log:    {'id': '10431597'
standard/log:    {'id': '10431784'
standard/log:    {'id': '10431864'
standard/log:    {'id': '10431878'
standard/log:    {'id': '10431952'
standard/log:    {'id': '10431962'
standard/log:    {'id': '10431987'
standard/log:    {'id': '10432011'
standard/log:    {'id': '10432023'
standard/log:    {'id': '10432089'
standard/log:    {'id': '10432123'
standard/log:    {'id': '10432130'
standard/log:    {'id': '10432133'
standard/log:    {'id': '10479610'
standard/log:    {'id': '10482724'
standard/log:    {'id': '10483560'
standard/log:    {'id': '10483578'
standard/log:    {'id': '10483621'
standard/log:    {'id': '10483741'
standard/log:    {'id': '10488902'
standard/log:    {'id': '10488960'
standard/log:    {'id': '10489211'
standard/log:    {'id': '10489230'
standard/log:    {'id': '10489670'
standard/log:    {'id': '10489735'
standard/log:    {'id': '10489806'
standard/log:    {'id': '10489841'
standard/log:    {'id': '10490038'
standard/log:    {'id': '10490168'
standard/log:    {'id': '10490170'
standard/log:    {'id': '10490208'
standard/log:    {'id': '10490231'
standard/log:    {'id': '10490256'
standard/log:    {'id': '10490257'
standard/log:    {'id': '10490259'
standard/log:    {'id': '10490268'
standard/log:    {'id': '10490269'
standard/log:    {'id': '10490271'
standard/log:    {'id': '10490272'
standard/log:    {'id': '10490274'
standard/log:    {'id': '10490275'
standard/log:    {'id': '10490277'
standard/log:    {'id': '10490278'
standard/log:    {'id': '10493209'
standard/log:    {'id': '10496349'
standard/log:    {'id': '10497676'
standard/log:    {'id': '10499131'

May 16 '20 16:05 trevorgokey

Those all seem complete on on the backend. Those just the gradient calculations, though. I have raised the limit on the nginx server. Hopefully that clears up the stale jobs, but is only a temporary fix.

May 16 '20 18:05 bennybp

Interesting. Here is a complete list from a start up to shutdown on between 200514 21:21:57 and 200516 00:45:34. I added the in the list points where it tried to push the jobs, where the error is returned, by #Tried:

May 16 '20 18:05 trevorgokey

Actually, I'll just attach said log file here qcf-stale.log

May 16 '20 18:05 trevorgokey

That stuff is all looking pretty normal. I don't think I will be able to absolutely figure out why these are so big without more instrumentation that catches large pushes and shows what is in them.

If you feel like giving it a shot, I think I know the easiest place to add it. If not, I will add debugging stuff to the next release.

May 17 '20 02:05 bennybp

Sure, I was thinking to just add some more prints in _request to see if I can pinpoint it. If you have some ideas I'd be happy to try them out.

May 17 '20 03:05 trevorgokey

ok that's actually what I was thinking. If you can check the size of the serialized payload and if greater than 50MB or so, dump the unserialized data to a file.

May 18 '20 14:05 bennybp

Yeah. Will do once I get more time on the cluster.. seemed to have run dry... will post again when I get some results. I can try other clusters to see if it appears.

That said, I did not get any more errors after the limit was raised, so it seems like to solved the immediate issue.

May 18 '20 21:05 trevorgokey

We are running our instances behind traefik now. I haven't seen these errors in a while.

Also, the managers in v0.50 are smarter and I think handle this more gracefully.

Sep 14 '23 16:09 bennybp

QCFractal QCFractal copied to clipboard

Completed tasks not being pushed because server returns Entity Too Large

QCFractal
QCFractal copied to clipboard