CDash icon indicating copy to clipboard operation
CDash copied to clipboard
verified publisher icon Kitware

Submission processing broke due locking problems / pid mismatch

Open estan opened this issue 5 years ago • 13 comments

Hi all,

We recently switched our Ubuntu CI build server from Ubuntu 18.04 to 20.04, and thus from CMake 3.10 to 3.16. Everything seemed to work great after the switch, and the first few submissions to our CDash server worked out fine.

However, today's CDash submissions were missing all the vital columns (Update, Configure, Build, Test) as well as the coverage rows in the coverage table on the dashboard.

When I looked into it, I discovered that something is going wrong in the processing of the submissions on the CDash side, so the submissions are not fully processed. The builds are not treated as "finished".

The submission upload for a such a build succeeds fine:

Request upload response: [{"status":0,"buildid":2397,"datafilesmd5":[0]}]
Curl response: [{"status":0}]
Upload file response: [{"status":0}]

But on the server side, in cdash.log, there are these types of errors:

[2020-05-03 19:00:15] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='11908' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","build_id":"2386","pid":11908} 
[2020-05-03 19:00:15] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","build_id":"2386","pid":11908} 
[2020-05-03 19:00:21] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='20115' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":20115} 
[2020-05-03 19:00:21] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":20115} 
[2020-05-03 19:00:25] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='4011' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":4011} 
[2020-05-03 19:00:25] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":4011} 
[2020-05-03 19:00:38] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='16320' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":16320} 
[2020-05-03 19:00:38] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":16320} 
[2020-05-03 19:00:39] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='17932' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","build_id":"2386","pid":17932} 
[2020-05-03 19:00:39] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","build_id":"2386","pid":17932} 
[2020-05-04 19:42:37] cdash.ERROR: 5 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":25282} 
[2020-05-04 19:42:37] cdash.ERROR: lock not released, unexpected pid mismatch: pid='18477' mypid='18785' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":18785} 
[2020-05-04 19:42:37] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":18785} 
[2020-05-04 19:42:37] cdash.ERROR: lock not released, unexpected pid mismatch: pid='18477' mypid='25282' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":25282} 
[2020-05-04 19:42:37] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":25282} 
[2020-05-04 19:42:37] cdash.ERROR: lock not released, unexpected pid mismatch: pid='18477' mypid='15609' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":15609} 
[2020-05-04 19:42:37] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":15609} 
[2020-05-04 19:42:37] cdash.ERROR: lock not released, unexpected pid mismatch: pid='18477' mypid='32536' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":32536} 
[2020-05-04 19:42:37] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":32536} 
[2020-05-04 20:10:38] cdash.ERROR: taking lock: projectid=1, other processor pid='18477' apparently stalled, lastupdated='2020-05-04 19:42:37' {"function":"AcquireProcessingLock","project_id":"1","pid":16623} 
[2020-05-04 20:10:38] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":6244} 
[2020-05-04 20:10:39] cdash.ERROR: lock not released, unexpected pid mismatch: pid='16623' mypid='16707' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":16707} 
[2020-05-04 20:10:39] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":16707} 
[2020-05-04 20:10:39] cdash.ERROR: lock not released, unexpected pid mismatch: pid='16623' mypid='6244' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":6244} 
[2020-05-04 20:10:39] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":6244} 
[2020-05-04 20:10:39] cdash.ERROR: lock not released, unexpected pid mismatch: pid='16623' mypid='28074' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":28074} 
[2020-05-04 20:10:39] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":28074} 
[2020-05-04 20:10:39] cdash.ERROR: lock not released, unexpected pid mismatch: pid='16623' mypid='18431' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":18431} 
[2020-05-04 20:10:39] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":18431} 
[2020-05-05 02:41:47] cdash.ERROR: taking lock: projectid=1, other processor pid='16623' apparently stalled, lastupdated='2020-05-04 20:10:38' {"function":"AcquireProcessingLock","project_id":"1","pid":14289} 
[2020-05-05 02:41:47] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":18901} 
[2020-05-05 02:41:47] cdash.ERROR: lock not released, unexpected pid mismatch: pid='14289' mypid='4295' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":4295} 
[2020-05-05 02:41:47] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":4295} 
[2020-05-05 02:41:47] cdash.ERROR: lock not released, unexpected pid mismatch: pid='14289' mypid='14265' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":14265} 
[2020-05-05 02:41:47] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":14265} 
[2020-05-05 02:41:47] cdash.ERROR: lock not released, unexpected pid mismatch: pid='14289' mypid='6208' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":6208} 
[2020-05-05 02:41:47] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":6208} 
[2020-05-05 02:41:47] cdash.ERROR: lock not released, unexpected pid mismatch: pid='14289' mypid='18901' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":18901} 
[2020-05-05 02:41:47] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":18901} 
[2020-05-05 06:55:33] cdash.ERROR: taking lock: projectid=1, other processor pid='14289' apparently stalled, lastupdated='2020-05-05 02:41:47' {"function":"AcquireProcessingLock","project_id":"1","pid":7104} 
[2020-05-05 06:55:33] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":12187} 
[2020-05-05 06:55:34] cdash.ERROR: lock not released, unexpected pid mismatch: pid='7104' mypid='12247' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":12247} 
[2020-05-05 06:55:34] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":12247} 
[2020-05-05 06:55:34] cdash.ERROR: lock not released, unexpected pid mismatch: pid='7104' mypid='9975' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":9975} 
[2020-05-05 06:55:34] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":9975} 
[2020-05-05 06:55:34] cdash.ERROR: lock not released, unexpected pid mismatch: pid='7104' mypid='4561' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":4561} 
[2020-05-05 06:55:34] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":4561} 
[2020-05-05 06:55:34] cdash.ERROR: lock not released, unexpected pid mismatch: pid='7104' mypid='12187' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":12187} 
[2020-05-05 06:55:34] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":12187} 
[2020-05-05 08:28:01] cdash.ERROR: taking lock: projectid=1, other processor pid='7104' apparently stalled, lastupdated='2020-05-05 06:55:33' {"function":"AcquireProcessingLock","project_id":"1","pid":7029} 
[2020-05-05 08:28:01] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":31519} 
[2020-05-05 08:28:01] cdash.ERROR: Too many attempts to process '/home/orexplore/www/CDash-2.6.0-prebuilt/backup/insight_buildbot-w-focal-1_b-insight-test-2394-clang-format-9-1-5-1-2-gb3b28d5_20200503-1852-Tests_158853242127.73_Done.xml' {"function":"ProcessSubmissions","project_id":"1","pid":6959} 
[2020-05-05 08:28:01] cdash.ERROR: Too many attempts to process '/home/orexplore/www/CDash-2.6.0-prebuilt/backup/insight_buildbot-w-focal-1_b-insight-test-2394-clang-format-9-1-5-1-2-gb3b28d5_20200503-1852-Tests_158853241543.23_Done.xml' {"function":"ProcessSubmissions","project_id":"1","pid":9964} 
[2020-05-05 08:28:01] cdash.ERROR: Too many attempts to process '/home/orexplore/www/CDash-2.6.0-prebuilt/backup/1164795271.xml' {"function":"ProcessSubmissions","project_id":"1","pid":21830} 
[2020-05-05 08:28:01] cdash.ERROR: Too many attempts to process '/home/orexplore/www/CDash-2.6.0-prebuilt/backup/insight_buildbot-w-focal-1_b-insight-test-2394-clang-format-9-1-5-1-2-gb3b28d5_20200503-1852-Tests_158853242501.28_Done.xml' {"function":"ProcessSubmissions","project_id":"1","pid":31519} 
[2020-05-05 08:28:02] cdash.ERROR: lock not released, unexpected pid mismatch: pid='7029' mypid='31519' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":31519} 
[2020-05-05 08:28:02] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":31519} 
[2020-05-05 08:28:02] cdash.ERROR: lock not released, unexpected pid mismatch: pid='7029' mypid='21830' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":21830} 
[2020-05-05 08:28:02] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":21830} 
[2020-05-05 08:28:03] cdash.ERROR: lock not released, unexpected pid mismatch: pid='7029' mypid='9964' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","build_id":"2387","pid":9964} 
[2020-05-05 08:28:03] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","build_id":"2387","pid":9964} 
[2020-05-05 08:28:03] cdash.ERROR: lock not released, unexpected pid mismatch: pid='7029' mypid='6959' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","build_id":"2387","pid":6959} 
[2020-05-05 08:28:03] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","build_id":"2387","pid":6959} 
[2020-05-05 08:56:05] cdash.ERROR: taking lock: projectid=1, other processor pid='7029' apparently stalled, lastupdated='2020-05-05 08:28:03' {"function":"AcquireProcessingLock","project_id":"1","pid":30937} 
[2020-05-05 08:56:06] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":17250} 
[2020-05-05 08:56:06] cdash.ERROR: lock not released, unexpected pid mismatch: pid='30937' mypid='12296' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":12296} 
[2020-05-05 08:56:06] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":12296} 
[2020-05-05 08:56:06] cdash.ERROR: lock not released, unexpected pid mismatch: pid='30937' mypid='30784' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":30784} 
[2020-05-05 08:56:06] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":30784} 
[2020-05-05 08:56:06] cdash.ERROR: lock not released, unexpected pid mismatch: pid='30937' mypid='25894' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":25894} 
[2020-05-05 08:56:06] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":25894} 
[2020-05-05 08:56:06] cdash.ERROR: lock not released, unexpected pid mismatch: pid='30937' mypid='17250' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":17250} 
[2020-05-05 08:56:06] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":17250} 
[2020-05-05 12:04:22] cdash.ERROR: taking lock: projectid=1, other processor pid='30937' apparently stalled, lastupdated='2020-05-05 08:56:06' {"function":"AcquireProcessingLock","project_id":"1","pid":527} 
[2020-05-05 12:04:22] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":561} 
[2020-05-05 12:04:23] cdash.ERROR: lock not released, unexpected pid mismatch: pid='527' mypid='15568' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":15568} 
[2020-05-05 12:04:23] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":15568} 
[2020-05-05 12:04:23] cdash.ERROR: lock not released, unexpected pid mismatch: pid='527' mypid='26902' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":26902} 
[2020-05-05 12:04:23] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":26902} 
[2020-05-05 12:04:23] cdash.ERROR: lock not released, unexpected pid mismatch: pid='527' mypid='31638' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":31638} 
[2020-05-05 12:04:23] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":31638} 
[2020-05-05 12:04:23] cdash.ERROR: lock not released, unexpected pid mismatch: pid='527' mypid='561' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":561} 
[2020-05-05 12:04:23] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":561} 
[2020-05-05 13:09:12] cdash.ERROR: taking lock: projectid=1, other processor pid='527' apparently stalled, lastupdated='2020-05-05 12:04:23' {"function":"AcquireProcessingLock","project_id":"1","pid":24412} 
[2020-05-05 13:09:13] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":27516} 
[2020-05-05 13:09:13] cdash.ERROR: lock not released, unexpected pid mismatch: pid='24412' mypid='26256' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":26256} 
[2020-05-05 13:09:13] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":26256} 
[2020-05-05 13:09:13] cdash.ERROR: lock not released, unexpected pid mismatch: pid='24412' mypid='27474' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":27474} 
[2020-05-05 13:09:13] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":27474} 
[2020-05-05 13:09:13] cdash.ERROR: lock not released, unexpected pid mismatch: pid='24412' mypid='29114' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":29114} 
[2020-05-05 13:09:13] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":29114} 
[2020-05-05 13:09:13] cdash.ERROR: lock not released, unexpected pid mismatch: pid='24412' mypid='27516' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":27516} 
[2020-05-05 13:09:13] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":27516} 
[2020-05-05 14:00:07] cdash.ERROR: taking lock: projectid=1, other processor pid='24412' apparently stalled, lastupdated='2020-05-05 13:09:13' {"function":"AcquireProcessingLock","project_id":"1","pid":22895} 
[2020-05-05 14:00:07] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":18314} 
[2020-05-05 14:00:07] cdash.ERROR: Too many attempts to process '/home/orexplore/www/CDash-2.6.0-prebuilt/backup/insight_buildbot-w-focal-1_b-insight-test-2394-clang-format-9-1-5-1-2-gb3b28d5_20200503-1852-Tests_158853243803.6_Done.xml' {"function":"ProcessSubmissions","project_id":"1","pid":18280} 
[2020-05-05 14:00:08] cdash.ERROR: lock not released, unexpected pid mismatch: pid='22895' mypid='25072' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":25072} 
[2020-05-05 14:00:08] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":25072} 
[2020-05-05 14:00:08] cdash.ERROR: lock not released, unexpected pid mismatch: pid='22895' mypid='25277' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":25277} 
[2020-05-05 14:00:08] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":25277} 
[2020-05-05 14:00:08] cdash.ERROR: lock not released, unexpected pid mismatch: pid='22895' mypid='18314' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":18314} 
[2020-05-05 14:00:08] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":18314} 
[2020-05-05 14:00:08] cdash.ERROR: lock not released, unexpected pid mismatch: pid='22895' mypid='18280' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":18280} 
[2020-05-05 14:00:08] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":18280} 
[2020-05-05 14:32:04] cdash.ERROR: taking lock: projectid=1, other processor pid='22895' apparently stalled, lastupdated='2020-05-05 14:00:08' {"function":"AcquireProcessingLock","project_id":"1","pid":20880} 
[2020-05-05 14:32:04] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":28362} 
[2020-05-05 14:32:04] cdash.ERROR: Too many attempts to process '/home/orexplore/www/CDash-2.6.0-prebuilt/backup/insight_buildbot-w-focal-1_b-insight-test-2394-clang-format-9-1-5-1-2-gb3b28d5_20200503-1852-Tests_158862135727.46_Done.xml' {"function":"ProcessSubmissions","project_id":"1","pid":30536} 
[2020-05-05 14:32:04] cdash.ERROR: Too many attempts to process '/home/orexplore/www/CDash-2.6.0-prebuilt/backup/insight_buildbot-w-focal-1_b-insight-test-2394-clang-format-9-1-5-1-2-gb3b28d5_20200503-1852-Tests_158862135727.46_Done_1.' {"function":"ProcessSubmissions","project_id":"1","pid":1000} 
[2020-05-05 14:32:04] cdash.ERROR: Too many attempts to process '/home/orexplore/www/CDash-2.6.0-prebuilt/backup/insight_buildbot-w-focal-1_b-insight-test-2394-clang-format-9-1-5-1-2-gb3b28d5_20200503-1852-Tests_158862135731.71_Done.xml' {"function":"ProcessSubmissions","project_id":"1","pid":30534} 
[2020-05-05 14:32:05] cdash.ERROR: lock not released, unexpected pid mismatch: pid='20880' mypid='28362' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":28362} 
[2020-05-05 14:32:05] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":28362} 
[2020-05-05 14:32:39] cdash.ERROR: lock not released, unexpected pid mismatch: pid='20880' mypid='30536' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","build_id":"2387","pid":30536} 
[2020-05-05 14:32:39] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","build_id":"2387","pid":30536} 
[2020-05-05 14:33:49] cdash.ERROR: lock not released, unexpected pid mismatch: pid='20880' mypid='30534' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","build_id":"2388","pid":30534} 
[2020-05-05 14:33:49] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","build_id":"2388","pid":30534} 
[2020-05-05 14:37:23] cdash.ERROR: lock not released, unexpected pid mismatch: pid='20880' mypid='1000' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":1000} 
[2020-05-05 14:37:23] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":1000} 
[2020-05-05 16:58:28] cdash.ERROR: taking lock: projectid=1, other processor pid='20880' apparently stalled, lastupdated='2020-05-05 14:37:23' {"function":"AcquireProcessingLock","project_id":"1","pid":11588} 
[2020-05-05 16:58:29] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":21278} 
[2020-05-05 16:58:29] cdash.ERROR: lock not released, unexpected pid mismatch: pid='11588' mypid='23819' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":23819} 
[2020-05-05 16:58:29] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":23819} 
[2020-05-05 16:58:29] cdash.ERROR: lock not released, unexpected pid mismatch: pid='11588' mypid='24772' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":24772} 
[2020-05-05 16:58:29] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":24772} 
[2020-05-05 16:58:29] cdash.ERROR: lock not released, unexpected pid mismatch: pid='11588' mypid='23816' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":23816} 
[2020-05-05 16:58:29] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":23816} 
[2020-05-05 16:58:29] cdash.ERROR: lock not released, unexpected pid mismatch: pid='11588' mypid='21278' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":21278} 
[2020-05-05 16:58:29] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":21278}

Note that even if the log excerpt here has errors dating back to May 3 and May 4, it is the builds on May 5 (today) that were lacking result data on the dashboard. I included the May 3 and May 4 output because I suspect something went wrong there which corrupted things, so that subsequent submissions could not be processed (?)

Has anyone had a similar issue, or know what I can do to recover?

Another thing which might have changed except for the CMake version in our migration to Ubuntu 20.04 is that we now use a more recent clang-tidy, and we get many (?) more clang-tidy warnings reported (thousands), in case that could be a clue.

We are using the prebuilt CDash 2.6.0, running with PHP 7.2 on Ubuntu 18.04.

estan avatar May 05 '20 18:05 estan

Did these recent upgrades affect the server where CDash is running, or only where CTest is building & testing your code?

zackgalbreath avatar May 05 '20 18:05 zackgalbreath

Did these recent upgrades affect the server where CDash is running, or only where CTest is building & testing your code?

@zackgalbreath Sorry, I should have been more clear about this: Nothing has changed on the server running CDash. It is still running on Ubuntu 18.04. The only thing that has changed is that our builder (the one building and submitting) is now running Ubuntu 20.04.

I had a look at the server APT log, and there has not been any change in the installed packages since yesterday (when we had builds that worked fine).

I should clarify more: The builds that did work fine yesterday were after our switch to 20.04 on the builder. So we had a few builds that worked fine, but then something broke today.

estan avatar May 05 '20 19:05 estan

Actually, looking back at the dashboard for yesterday, there was one build which worked OK, and then another which did not. The dashboard for yesterday looks like:

may4dashboard

The b-insight-test-2-clang-format-9-1.5.1-2-gb3b28d5 build (May 4, 2020 - 19:39 UTC) worked fine, but the b-insight-test-3-clang-format-9-1.5.1-2-gb3b28d5 build (May 4, 2020 - 20:04 UTC) is incompletely processed, so missing result data.

Below are the logs from the CTest run for these two builds:

estan avatar May 05 '20 19:05 estan

If it helps in correlating these builds with the output in cdash.log: The first build above (that worked fine) is CDash buildid 2387, the second one (that broke) is buildid 2388.

estan avatar May 05 '20 19:05 estan

Hmm, I'm afraid that I'm not entirely sure what's going wrong, but here's a couple things you can try.

  1. You can force CDash's asynchronous processing loop to (re)start by visiting https://your-cdash-url/ajax/processsubmissions.php?projectid=1&force=1 with a web browser. I'd be curious to know if this allows the stuck builds to finish processing or if CDash runs into the same errors over & over again.

  2. To workaround this error: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() You could manually edit include/submissions_functions.php to wrap the definition of ProcessSubmissionsErrorHandler() in a if() block like so:

if (!function_exists('ProcessSubmissionsErrorHandler')) {
    function ProcessSubmissionsErrorHandler($projectid)
    {
    ....
    }
}

That probably won't fix the problem, but it might allow some more useful error messages to wind up in the log.

zackgalbreath avatar May 06 '20 17:05 zackgalbreath

Thanks for the tips @zackgalbreath, I will investigate. If all else fails, I'll try a re-installation + fresh database. Maintaining the build history is not crucial to us, so it would be acceptable.

estan avatar May 07 '20 06:05 estan

  1. You can force CDash's asynchronous processing loop to (re)start by visiting https://your-cdash-url/ajax/processsubmissions.php?projectid=1&force=1 with a web browser. I'd be curious to know if this allows the stuck builds to finish processing or if CDash runs into the same errors over & over again.

I gave this a shot:

Browser Output

begin processSubmissions.php
no argc, context is web browser or some other non-command-line...
projectid='1'
pid='24942'
force='1'
AcquireProcessingLock returned true
Done with ResetApparentlyStalledSubmissions
ProcessSubmissionsErrorHandler: ReleasedProcessingLock(1) returned true

cdash.log

[2020-05-07 08:47:33] cdash.ERROR: 4 submission records assumed stalled, reset to status=0 {"function":"ResetApparentlyStalledSubmissions","project_id":"1","pid":24942} 
[2020-05-07 08:47:33] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":24942}

Doing it again gave:

Browser Output

begin processSubmissions.php
no argc, context is web browser or some other non-command-line...
projectid='1'
pid='5353'
force='1'
AcquireProcessingLock returned true
Done with ResetApparentlyStalledSubmissions
ProcessSubmissionsErrorHandler: ReleasedProcessingLock(1) returned true

cdash.log

[2020-05-07 08:51:11] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":5353}

(No 4 submission records assumed stalled message this second time)

So it seems it's getting the same error. I will try your second suggestion next.

estan avatar May 07 '20 08:05 estan

  1. To workaround this error: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() You could manually edit include/submissions_functions.php to wrap the definition of ProcessSubmissionsErrorHandler() in a if() block like so:
if (!function_exists('ProcessSubmissionsErrorHandler')) {
    function ProcessSubmissionsErrorHandler($projectid)
    {
    ....
    }
}

That probably won't fix the problem, but it might allow some more useful error messages to wind up in the log.

I had a go at this. I had to wrap all functions in that file similarly.

The browser request is in progress, and it seems like it is crawling through the backlog of submissions. The first failing one from May 4 now has data on the dashboard, as does a couple of ones from May 5 :thumbsup:

Many thanks for the help. I'll let the browser request finish and report back on the end result, and see whether new incoming builds will finish successfully.

estan avatar May 07 '20 09:05 estan

It is still trotting along, and large parts of the backlog of builds are now processed. I have gotten a single:

[2020-05-07 09:12:34] cdash.ERROR: lock not released, unexpected pid mismatch: pid='18056' mypid='19575' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":19575}

in the log.

Heading out for lunch. I should have a final verdict on the result after that.

estan avatar May 07 '20 09:05 estan

I can report that the browser request now finished with output:

begin processSubmissions.php
no argc, context is web browser or some other non-command-line...
projectid='1'
pid='18056'
force='1'
AcquireProcessingLock returned true
Done with ResetApparentlyStalledSubmissions

  OK
  


  OK
  

<and so on>


  OK
  


  OK
  

Done with ProcessSubmissions
Done with DeleteOldSubmissionRecords
ReleasedProcessingLock returned true
end processSubmissions.php

Looking at the dashboard, all but a few of the builds are now fully processed. I will look into those few, it may be that they are builds that were manually cancelled on our Buildbot mid-way through.

estan avatar May 07 '20 11:05 estan

Doing the request again, a few more finished (6 more "OK" printouts), but now it seems to be a no-op when I do it again.

There is a single build which is missing the Build, Test and Coverage results (but does have the Configure results) that I can't quite explain. The CTest output for that build looks normal. I'll just ignore that one for now, and try a fresh push to our repo and see if the CDash submission is OK again.

estan avatar May 07 '20 11:05 estan

A re-triggered build was submitted successfully, with no missing result on the dashboard.

I then tried removing the if (!function_exists(..)) guards in include/submission_functions.php and trigger another build. That one was also processed fully and appearing fine in the dashboard. The cdash.log had this in it though:

[2020-05-07 11:21:24] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='5873' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":5873} 
[2020-05-07 11:21:24] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":5873} 
[2020-05-07 11:21:28] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='5873' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":5873} 
[2020-05-07 11:21:28] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":5873} 
[2020-05-07 11:21:31] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='31467' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","build_id":"2417","pid":31467} 
[2020-05-07 11:21:31] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","build_id":"2417","pid":31467} 
[2020-05-07 11:21:46] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='1066' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","pid":1066} 
[2020-05-07 11:21:46] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","pid":1066} 
[2020-05-07 11:21:48] cdash.ERROR: lock not released, unexpected pid mismatch: pid='0' mypid='31768' - attempt to unlock a lock we don't own... {"function":"ReleaseProcessingLock","project_id":"1","build_id":"2417","pid":31768} 
[2020-05-07 11:21:48] cdash.ERROR: Fatal error:Cannot redeclare ProcessSubmissionsErrorHandler() (previously declared in /home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php:392) {"function":"/home/orexplore/www/CDash-2.6.0-prebuilt/include/submission_functions.php (392)","project_id":"1","build_id":"2417","pid":31768}

So problem seems sort of solved, though those ERRORs are a perhaps something to look into.

estan avatar May 07 '20 11:05 estan

Hm, it seems builds are again incompletely processed:

incomplete-builds

It seems that as soon as I've triggered those errors in https://github.com/Kitware/CDash/issues/987#issuecomment-625196151 once, then subsequent builds will not be fully processed.

I'll add back the if (!function_exists(..)) guards and manually process builds again, to get it back into working order.

So seems there is something to fix here :/

estan avatar May 07 '20 11:05 estan