MacOS 11.6.1 fails to submit large job logs > 20MB

[emmenlau]

I'm using CDash 3.0.3 on a self-hosted server in our internal network. On the same network (even the same network switch), I run a number of CI computers. All of them used to submit the jobs logs successfully to CDash for over a year.

Now since I upgraded the MacOS server to 11.6, it started failing to submit larger job logs. It does not always fail. Typically it works after a reboot for the better part of a week, then starts failing. After the first fail, most job submissions fail when the log is large. Our logs are often in the range of 20MB or more.

I run ctest from cmake 3.22.0, with options --tests-regex .*(Test|Check).* --parallel 8 --extra-verbose --test-model Experimental --test-action Test --test-action Submit

Here is the output I get from ctest:

[...]
100% tests passed, 0 tests failed out of 46
Total Test time (real) = 391.93 sec
UpdateCTestConfiguration  from :/System/Volumes/Data/data/ci-tmp-m1015Iv64c1300/Debug/TestProject/DartConfiguration.tcl
Parse Config file:/System/Volumes/Data/data/ci-tmp-m1015Iv64c1300/Debug/TestProject/DartConfiguration.tcl
Submit files
   Send to group: Experimental
   SubmitURL: http://cdash.intern.biodataanalysis.de/submit.php?project=TestProject
   Upload file: /System/Volumes/Data/data/ci-tmp-m1015Iv64c1300/Debug/TestProject/Testing/20220111-0019/Configure.xml to http://cdash.intern.biodataanalysis.de/submit.php?project=TestProject&FileName=MacOS%2020.6.0%20(aidan11.local)___master%20(250f1585)___20220111-0019-Experimental___XML___Configure.xml&build=master%20%28250f1585%29&site=MacOS%2020.6.0%20%28aidan11.local%29&stamp=20220111-0019-Experimental&MD5=1e887491ebd568524133b0ed55ae3588 Size: 6651
   Uploaded: /System/Volumes/Data/data/ci-tmp-m1015Iv64c1300/Debug/TestProject/Testing/20220111-0019/Configure.xml
   Upload file: /System/Volumes/Data/data/ci-tmp-m1015Iv64c1300/Debug/TestProject/Testing/20220111-0019/Build.xml to http://cdash.intern.biodataanalysis.de/submit.php?project=TestProject&FileName=MacOS%2020.6.0%20(aidan11.local)___master%20(250f1585)___20220111-0019-Experimental___XML___Build.xml&build=master%20%28250f1585%29&site=MacOS%2020.6.0%20%28aidan11.local%29&stamp=20220111-0019-Experimental&MD5=52c40edb01098eb8f9398902d6b36dba Size: 25152086
   Submit failed, waiting 5 seconds...
   Retry submission: Attempt 1 of 3
   Submit failed, waiting 5 seconds...
   Retry submission: Attempt 2 of 3
   Submit failed, waiting 5 seconds...
   Retry submission: Attempt 3 of 3
   Error when uploading file: /System/Volumes/Data/data/ci-tmp-m1015Iv64c1300/Debug/TestProject/Testing/20220111-0019/Build.xml
   Error message was: Operation too slow. Less than 1 bytes/sec transferred the last 120 seconds
   Problems when submitting via HTTP
Errors while running CTest
Output from these tests are in: /System/Volumes/Data/data/ci-tmp-m1015Iv64c1300/Debug/TestProject/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

[emmenlau]

To reduce the stress on this, I could find a cause for a major part of the warnings in my build, and reduce the log file size to about 1/3. Sine then the problem has not come up again...

[zackgalbreath]

This might be a good candidate for asychronous submission processing. Try setting $CDASH_ASYNCHRONOUS_SUBMISSION = true; in app/cdash/config/config.local.php to see if that improves your CDash's installation to parse large files.