ganga
ganga copied to clipboard
Published
20 hours ago •
ganga-devs
ganga-devs
Impossible to exit ganga session normally if queue is not completely empty
For the last couple of years, I find it impossible to exit a ganga session if the job queues are not completely empty. When I confirm that I want to force the exit, ganga seems to wait indefinitely for those tasks (I never had the patience to wait for longer than 20min). Then I use Ctrl+C and ganga tells me that I have to wait for at least "a while" before using Ctrl+C. After a restart, the job statuses are indeed all corrupted (e.g. in the "new" or "submitted" state) but seem to restore themselves when I print them.
If I choose "no" (do not force the exit), I am stuck in an infinite loop where ganga asks the same question again and again.
What is the actual minimum time I need to wait so that my jobs do not get seriously corrupted?
In my experience, this is almost guaranteed to happen if one or more subjobs are being finalised, even though the finalisation itself takes just a couple of seconds. I don't know whether more job finalisations are scheduled while exiting, but this type of job almost always seems to block the exiting process. There is no traceback after hitting Ctrl+C. In the example above, job finalisations were not running
Sorry, I misunderstood the word task here. So by queues here you mean the queues that Ganga use internally for downloading output from Dirac, or is this items that you put in the queues yourself. Can you try to type queues and post the output before you have this problem (I know it is too late when you see the problem, but it sounds as if it is pretty reproducible for you).
Yes, I mean the ganga monitoring threads. Usually, when ganga prints the warning message after the Ctrl+C event, it enumerates its own internal processes which were apparently still running. Yes, I could do that next time
As another thing, if you can force this issue with a session started as ganga --debug, it would be great. Just share the .ganga.log file with us afterwards.
Ok, I have some more processing to do, so I should be able to reproduce it in the next days in the debug mode
I was able to reproduce it immediately: I submitted 40 small jobs, saw that one single job was finalising and tried to exit ganga, this was at 11:49. At 12:16 I hit Ctrl+C and ganga exited immediately, this time without a warning or error. The whole time, I got debug outputs, but it looks like only the most recent events are written to the log:
2021-02-24 12:16:39,461 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::_checkActiveBackends:1198 DEBUG : Active Backends: {"Dirac" : [589, 612, 613, 614, 615, ], }
2021-02-24 12:16:39,489 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :332 DEBUG : *: backend=Dirac, isOwner=False, joblist=[589, 612, 613, 614, 615], queue=3
2021-02-24 12:16:39,490 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :336 DEBUG : Lock acquire successful. Updating jSet ['612', '589', '615', '614', '613'] with ['589', '612', '613', '614', '615'].
2021-02-24 12:16:39,490 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :342 DEBUG : Dirac backend job set exists. Added ['589', '612', '613', '614', '615'] to it.
2021-02-24 12:16:39,490 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :353 DEBUG : **: backend=Dirac, isOwner=False, joblist=['589', '612', '613', '614', '615'], queue=3
2021-02-24 12:16:39,490 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::_checkActiveBackends:1218 DEBUG : jList: ['589', '612', '613', '614', '615']
2021-02-24 12:16:39,491 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :647 DEBUG :
Processing Function: diskSpaceCheckJobInsertor
2021-02-24 12:16:39,491 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :655 DEBUG : cbHookEntry.enabled: False
2021-02-24 12:16:39,491 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :656 DEBUG : (time.time() - cbHookEntry._lastRun): 1694.143264055252
2021-02-24 12:16:39,491 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :657 DEBUG : cbHookEntry.timeout: 30
2021-02-24 12:16:39,491 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :647 DEBUG :
Processing Function: timeoutCheck
2021-02-24 12:16:39,491 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :655 DEBUG : cbHookEntry.enabled: True
2021-02-24 12:16:39,492 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :656 DEBUG : (time.time() - cbHookEntry._lastRun): 2.1935901641845703
2021-02-24 12:16:39,492 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :657 DEBUG : cbHookEntry.timeout: 0
2021-02-24 12:16:39,492 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :660 DEBUG : Running monitoring callback hook function timeoutCheck(**{'thisDict': <GangaCore.Utility.threads.SynchronisedObject object at 0x7fe2243983c8>})
2021-02-24 12:16:39,492 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :676 DEBUG :
Running runClientCallbacks
2021-02-24 12:16:39,492 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :679 DEBUG : Finished runClientCallbacks
2021-02-24 12:16:39,492 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :600 DEBUG : Finished Step
2021-02-24 12:16:39,492 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :605 DEBUG : Wait Condition
2021-02-24 12:16:40,498 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :605 DEBUG : Wait Condition
2021-02-24 12:16:41,502 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :619 DEBUG : Run on Demand
2021-02-24 12:16:41,503 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :579 DEBUG : Monitoring Loop is alive
2021-02-24 12:16:41,503 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :583 DEBUG : Monitoring loop lock acquired. Running loop: Steps -1
2021-02-24 12:16:41,503 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :597 DEBUG : Launching Monitoring Step
2021-02-24 12:16:41,503 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :647 DEBUG :
Processing Function: _checkActiveBackends
2021-02-24 12:16:41,503 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :655 DEBUG : cbHookEntry.enabled: True
2021-02-24 12:16:41,504 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :656 DEBUG : (time.time() - cbHookEntry._lastRun): 2.012883424758911
2021-02-24 12:16:41,504 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :657 DEBUG : cbHookEntry.timeout: 0
2021-02-24 12:16:41,504 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :660 DEBUG : Running monitoring callback hook function _checkActiveBackends(**{'thisMonitor': <JobRegistry_Monitor(GANGA_Update_Thread_JobRegistry_Monitor, started daemon 140605476824832)>, 'activeBackendsFunc': <bound method JobRegistry_Monitor.__defaultActiveBackendsFunc of <JobRegistry_Monitor(GANGA_Update_Thread_JobRegistry_Monitor, started daemon 140605476824832)>>, 'jobSlice': None})
2021-02-24 12:16:41,504 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::_checkActiveBackends:1185 DEBUG : calling function _checkActiveBackends
2021-02-24 12:16:41,504 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__defaultActiveBackendsFunc:1012 DEBUG : __defaultActiveBackendsFunc
2021-02-24 12:16:41,504 GANGA_Update_Thread_JobRegistry_Monitor Registry ::keys :377 DEBUG : keys
2021-02-24 12:16:41,505 GANGA_Update_Thread_JobRegistry_Monitor Registry ::ids :359 DEBUG : ids
2021-02-24 12:16:41,505 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__defaultActiveBackendsFunc:1027 DEBUG : Running over fixed_ids: [580, 581, 586, 587, 588, 589, 590, 593, 594, 606, 612, 613, 614, 615]
2021-02-24 12:16:41,505 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,505 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,505 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,506 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,506 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,506 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,507 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,507 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,507 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,508 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,508 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,508 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,508 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,509 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:41,509 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__defaultActiveBackendsFunc:1061 DEBUG : Returning active_backends: {"Dirac" : [589, 612, 613, 614, 615, ], }
2021-02-24 12:16:41,509 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::_checkActiveBackends:1198 DEBUG : Active Backends: {"Dirac" : [589, 612, 613, 614, 615, ], }
2021-02-24 12:16:41,510 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :332 DEBUG : *: backend=Dirac, isOwner=False, joblist=[589, 612, 613, 614, 615], queue=3
2021-02-24 12:16:41,510 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :336 DEBUG : Lock acquire successful. Updating jSet ['612', '589', '615', '614', '613'] with ['589', '612', '613', '614', '615'].
2021-02-24 12:16:41,510 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :342 DEBUG : Dirac backend job set exists. Added ['589', '612', '613', '614', '615'] to it.
2021-02-24 12:16:41,511 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :353 DEBUG : **: backend=Dirac, isOwner=False, joblist=['589', '612', '613', '614', '615'], queue=3
2021-02-24 12:16:41,511 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::_checkActiveBackends:1218 DEBUG : jList: ['589', '612', '613', '614', '615']
2021-02-24 12:16:41,511 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :647 DEBUG :
Processing Function: diskSpaceCheckJobInsertor
2021-02-24 12:16:41,511 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :655 DEBUG : cbHookEntry.enabled: False
2021-02-24 12:16:41,511 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :656 DEBUG : (time.time() - cbHookEntry._lastRun): 1696.1637394428253
2021-02-24 12:16:41,512 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :657 DEBUG : cbHookEntry.timeout: 30
2021-02-24 12:16:41,512 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :647 DEBUG :
Processing Function: timeoutCheck
2021-02-24 12:16:41,512 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :655 DEBUG : cbHookEntry.enabled: True
2021-02-24 12:16:41,512 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :656 DEBUG : (time.time() - cbHookEntry._lastRun): 2.01997709274292
2021-02-24 12:16:41,512 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :657 DEBUG : cbHookEntry.timeout: 0
2021-02-24 12:16:41,512 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :660 DEBUG : Running monitoring callback hook function timeoutCheck(**{'thisDict': <GangaCore.Utility.threads.SynchronisedObject object at 0x7fe2243983c8>})
2021-02-24 12:16:41,513 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :676 DEBUG :
Running runClientCallbacks
2021-02-24 12:16:41,513 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :679 DEBUG : Finished runClientCallbacks
2021-02-24 12:16:41,513 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :600 DEBUG : Finished Step
2021-02-24 12:16:41,513 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :605 DEBUG : Wait Condition
2021-02-24 12:16:41,946 GANGA_Update_Thread_JobRegistryFlusher Registry ::run :270 DEBUG : Auto-flushing: templates
2021-02-24 12:16:41,947 GANGA_Update_Thread_JobRegistryFlusher Registry ::values :385 DEBUG : values
2021-02-24 12:16:41,947 GANGA_Update_Thread_JobRegistryFlusher Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:41,948 GANGA_Update_Thread_JobRegistryFlusher Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,079 GANGA_Update_Thread_TaskRegistryFlusher Registry ::run :270 DEBUG : Auto-flushing: tasks
2021-02-24 12:16:42,079 GANGA_Update_Thread_TaskRegistryFlusher Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,517 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :605 DEBUG : Wait Condition
2021-02-24 12:16:42,855 Dummy-42 __init__ ::shutDownQueues :40 WARNING : Error in shutting down queues thread. Likely harmless
2021-02-24 12:16:42,855 Dummy-42 ShutdownManager ::_unprotected_ganga_exitfuncs:120 INFO : Shutting Down Ganga Repositories
2021-02-24 12:16:42,857 Dummy-42 Repository_runtime ::shutdown :170 INFO : Registry Shutdown
2021-02-24 12:16:42,858 Dummy-42 Repository_runtime ::flush_all :236 DEBUG : Flushing All repositories
2021-02-24 12:16:42,858 Dummy-42 Repository_runtime ::flush_all :242 DEBUG : Flushing: prep
2021-02-24 12:16:42,858 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,858 Dummy-42 Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:42,859 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,859 Dummy-42 Repository_runtime ::flush_all :242 DEBUG : Flushing: jobs
2021-02-24 12:16:42,859 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,860 Dummy-42 Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:42,860 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,860 Dummy-42 Repository_runtime ::flush_all :242 DEBUG : Flushing: templates
2021-02-24 12:16:42,860 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,861 Dummy-42 Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:42,861 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,861 Dummy-42 Repository_runtime ::flush_all :242 DEBUG : Flushing: box
2021-02-24 12:16:42,861 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,861 Dummy-42 Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:42,862 Dummy-42 Repository_runtime ::flush_all :242 DEBUG : Flushing: tasks
2021-02-24 12:16:42,862 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,862 Dummy-42 Repository_runtime ::shutdown :179 DEBUG : ['prep', 'jobs', 'templates', 'box', 'tasks']
2021-02-24 12:16:42,864 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:42,864 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:42,864 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,864 Dummy-42 Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:42,865 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,866 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:42,866 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:42,866 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:42,868 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: prep.metadata
2021-02-24 12:16:42,868 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 0
2021-02-24 12:16:42,877 Dummy-42 GangaRepositoryXML ::index_write :363 DEBUG : Writing: ('sharerefs', 'ShareRef', {})
2021-02-24 12:16:42,890 Dummy-42 Objects ::_index_cache :1011 DEBUG : Warning: Setting IndexCache data on live object, please avoid!
2021-02-24 12:16:43,202 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 0
2021-02-24 12:16:43,229 Dummy-42 GangaThread ::stop :59 DEBUG : Stopping: SessionLockRefresher
2021-02-24 12:16:43,230 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: prep
2021-02-24 12:16:43,290 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:43,290 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:43,291 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,291 Dummy-42 Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:43,291 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,291 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:43,291 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:43,291 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,292 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: jobs.metadata
2021-02-24 12:16:43,317 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: jobs
2021-02-24 12:16:43,317 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 589
2021-02-24 12:16:43,325 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #0
2021-02-24 12:16:43,325 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #1
2021-02-24 12:16:43,325 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #2
2021-02-24 12:16:43,325 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #3
2021-02-24 12:16:43,325 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #4
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #5
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #6
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #7
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #8
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #9
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #10
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #11
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #12
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #13
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #14
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #15
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #16
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #17
2021-02-24 12:16:43,326 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #18
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #19
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #20
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #21
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #22
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #23
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #24
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #25
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #26
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #27
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #28
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #29
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #30
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #31
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #32
2021-02-24 12:16:43,327 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #33
2021-02-24 12:16:43,328 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #34
2021-02-24 12:16:43,328 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #35
2021-02-24 12:16:43,328 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #36
2021-02-24 12:16:43,328 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #37
2021-02-24 12:16:43,328 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #0
2021-02-24 12:16:43,328 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #1
2021-02-24 12:16:43,329 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #2
2021-02-24 12:16:43,329 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #3
2021-02-24 12:16:43,330 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #4
2021-02-24 12:16:43,330 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #5
2021-02-24 12:16:43,330 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #6
2021-02-24 12:16:43,331 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #7
2021-02-24 12:16:43,331 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #8
2021-02-24 12:16:43,332 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #9
2021-02-24 12:16:43,332 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #10
2021-02-24 12:16:43,332 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #11
2021-02-24 12:16:43,333 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #12
2021-02-24 12:16:43,333 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #13
2021-02-24 12:16:43,334 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #14
2021-02-24 12:16:43,334 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #15
2021-02-24 12:16:43,334 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #16
2021-02-24 12:16:43,335 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #17
2021-02-24 12:16:43,335 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #18
2021-02-24 12:16:43,336 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #19
2021-02-24 12:16:43,336 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #20
2021-02-24 12:16:43,337 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #21
2021-02-24 12:16:43,337 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #22
2021-02-24 12:16:43,337 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #23
2021-02-24 12:16:43,338 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #24
2021-02-24 12:16:43,338 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #25
2021-02-24 12:16:43,339 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #26
2021-02-24 12:16:43,339 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #27
2021-02-24 12:16:43,340 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #28
2021-02-24 12:16:43,340 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #29
2021-02-24 12:16:43,340 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #30
2021-02-24 12:16:43,341 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #31
2021-02-24 12:16:43,341 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #32
2021-02-24 12:16:43,342 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #33
2021-02-24 12:16:43,342 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #34
2021-02-24 12:16:43,342 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #35
2021-02-24 12:16:43,343 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #36
2021-02-24 12:16:43,343 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #37
2021-02-24 12:16:43,372 Dummy-42 GangaRepositoryXML ::index_write :363 DEBUG : Writing: ('jobs', 'Job', {'status': 'submitted', 'id': 589, 'name': 'Bd2JPsiKstar-2016-26-Sim09b-MagDown-DST-1614039133', 'comment': '', 'display:fqid': '589', 'display:status': 'submitted', 'display:name': 'Bd2JPsiKstar-2016-26-Sim09b-MagDown-DST-1614039133', 'display:subjobs': '38', 'display:application': 'GaudiExec', 'display:backend': 'Dirac', 'display:backend.actualCE': 'None', 'display:comment': '', 'display:subjob status': '0/0/0/38', 'subjobs:status': ['completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed', 'completed']})
2021-02-24 12:16:43,376 Dummy-42 Objects ::_index_cache :1011 DEBUG : Warning: Setting IndexCache data on live object, please avoid!
2021-02-24 12:16:43,376 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 612
2021-02-24 12:16:43,385 Dummy-42 GangaRepositoryXML ::index_write :363 DEBUG : Writing: ('jobs', 'Job', {'status': 'submitted', 'id': 612, 'name': 'Bs2DsPi-2016-26-Sim09b-MagDown-DST-1614162295', 'comment': '', 'display:fqid': '612', 'display:status': 'submitted', 'display:name': 'Bs2DsPi-2016-26-Sim09b-MagDown-DST-1614162295', 'display:subjobs': '10', 'display:application': 'GaudiExec', 'display:backend': 'Dirac', 'display:backend.actualCE': 'None', 'display:comment': '', 'display:subjob status': '0/0/1/8', 'subjobs:status': ['completed', 'completed', 'completed', 'completing', 'completed', 'completed', 'completed', 'completed', 'completed', 'submitted']})
2021-02-24 12:16:43,403 Dummy-42 Objects ::_index_cache :1011 DEBUG : Warning: Setting IndexCache data on live object, please avoid!
2021-02-24 12:16:43,403 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 613
2021-02-24 12:16:43,415 Dummy-42 GangaRepositoryXML ::index_write :363 DEBUG : Writing: ('jobs', 'Job', {'status': 'submitted', 'id': 613, 'name': 'Bs2DsPi-2016-26-Sim09b-MagUp-DST-1614162999', 'comment': '', 'display:fqid': '613', 'display:status': 'submitted', 'display:name': 'Bs2DsPi-2016-26-Sim09b-MagUp-DST-1614162999', 'display:subjobs': '10', 'display:application': 'GaudiExec', 'display:backend': 'Dirac', 'display:backend.actualCE': 'None', 'display:comment': '', 'display:subjob status': '2/0/0/0', 'subjobs:status': ['submitted', 'submitted', 'running', 'submitted', 'submitted', 'submitted', 'running', 'submitted', 'submitted', 'submitted']})
2021-02-24 12:16:43,419 Dummy-42 Objects ::_index_cache :1011 DEBUG : Warning: Setting IndexCache data on live object, please avoid!
2021-02-24 12:16:43,419 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 614
2021-02-24 12:16:43,451 Dummy-42 GangaRepositoryXML ::index_write :363 DEBUG : Writing: ('jobs', 'Job', {'status': 'submitted', 'id': 614, 'name': 'Bs2DsPi-2016-28r1-Sim09h-MagDown-LDST-1614163386', 'comment': '', 'display:fqid': '614', 'display:status': 'submitted', 'display:name': 'Bs2DsPi-2016-28r1-Sim09h-MagDown-LDST-1614163386', 'display:subjobs': '10', 'display:application': 'GaudiExec', 'display:backend': 'Dirac', 'display:backend.actualCE': 'None', 'display:comment': '', 'display:subjob status': '0/0/0/0', 'subjobs:status': ['submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted']})
2021-02-24 12:16:43,456 Dummy-42 Objects ::_index_cache :1011 DEBUG : Warning: Setting IndexCache data on live object, please avoid!
2021-02-24 12:16:43,456 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 615
2021-02-24 12:16:43,462 Dummy-42 GangaRepositoryXML ::index_write :363 DEBUG : Writing: ('jobs', 'Job', {'status': 'submitted', 'id': 615, 'name': 'Bs2DsPi-2016-28r1-Sim09h-MagUp-LDST-1614163567', 'comment': '', 'display:fqid': '615', 'display:status': 'submitted', 'display:name': 'Bs2DsPi-2016-28r1-Sim09h-MagUp-LDST-1614163567', 'display:subjobs': '10', 'display:application': 'GaudiExec', 'display:backend': 'Dirac', 'display:backend.actualCE': 'None', 'display:comment': '', 'display:subjob status': '0/0/0/0', 'subjobs:status': ['submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted', 'submitted']})
2021-02-24 12:16:43,473 Dummy-42 Objects ::_index_cache :1011 DEBUG : Warning: Setting IndexCache data on live object, please avoid!
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #0
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #1
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #2
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #3
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #4
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #5
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #6
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #7
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #8
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #9
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #10
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #11
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #12
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #13
2021-02-24 12:16:43,475 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #14
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #15
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #16
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #17
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #18
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #19
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #20
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #21
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #22
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #23
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #24
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #25
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #26
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #27
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #28
2021-02-24 12:16:43,476 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #29
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #30
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #31
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #32
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #33
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #34
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #35
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #36
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #37
2021-02-24 12:16:43,477 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #0
2021-02-24 12:16:43,478 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #1
2021-02-24 12:16:43,478 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #2
2021-02-24 12:16:43,479 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #3
2021-02-24 12:16:43,479 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #4
2021-02-24 12:16:43,479 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #5
2021-02-24 12:16:43,480 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #6
2021-02-24 12:16:43,480 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #7
2021-02-24 12:16:43,481 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #8
2021-02-24 12:16:43,481 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #9
2021-02-24 12:16:43,482 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #10
2021-02-24 12:16:43,482 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #11
2021-02-24 12:16:43,482 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #12
2021-02-24 12:16:43,483 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #13
2021-02-24 12:16:43,483 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #14
2021-02-24 12:16:43,484 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #15
2021-02-24 12:16:43,484 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #16
2021-02-24 12:16:43,484 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #17
2021-02-24 12:16:43,485 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #18
2021-02-24 12:16:43,485 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #19
2021-02-24 12:16:43,486 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #20
2021-02-24 12:16:43,486 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #21
2021-02-24 12:16:43,487 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #22
2021-02-24 12:16:43,487 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #23
2021-02-24 12:16:43,487 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #24
2021-02-24 12:16:43,488 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #25
2021-02-24 12:16:43,488 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #26
2021-02-24 12:16:43,489 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #27
2021-02-24 12:16:43,489 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #28
2021-02-24 12:16:43,489 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #29
2021-02-24 12:16:43,490 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #30
2021-02-24 12:16:43,491 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #31
2021-02-24 12:16:43,491 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #32
2021-02-24 12:16:43,491 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #33
2021-02-24 12:16:43,492 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #34
2021-02-24 12:16:43,492 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #35
2021-02-24 12:16:43,493 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #36
2021-02-24 12:16:43,493 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #37
2021-02-24 12:16:43,522 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :619 DEBUG : Run on Demand
2021-02-24 12:16:43,523 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :579 DEBUG : Monitoring Loop is alive
2021-02-24 12:16:43,523 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :583 DEBUG : Monitoring loop lock acquired. Running loop: Steps -1
2021-02-24 12:16:43,523 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :597 DEBUG : Launching Monitoring Step
2021-02-24 12:16:43,523 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :647 DEBUG :
Processing Function: _checkActiveBackends
2021-02-24 12:16:43,523 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :655 DEBUG : cbHookEntry.enabled: True
2021-02-24 12:16:43,523 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :656 DEBUG : (time.time() - cbHookEntry._lastRun): 2.012115716934204
2021-02-24 12:16:43,523 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :657 DEBUG : cbHookEntry.timeout: 0
2021-02-24 12:16:43,523 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :660 DEBUG : Running monitoring callback hook function _checkActiveBackends(**{'thisMonitor': <JobRegistry_Monitor(GANGA_Update_Thread_JobRegistry_Monitor, started daemon 140605476824832)>, 'activeBackendsFunc': <bound method JobRegistry_Monitor.__defaultActiveBackendsFunc of <JobRegistry_Monitor(GANGA_Update_Thread_JobRegistry_Monitor, started daemon 140605476824832)>>, 'jobSlice': None})
2021-02-24 12:16:43,524 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::_checkActiveBackends:1185 DEBUG : calling function _checkActiveBackends
2021-02-24 12:16:43,524 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__defaultActiveBackendsFunc:1012 DEBUG : __defaultActiveBackendsFunc
2021-02-24 12:16:43,526 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 589
2021-02-24 12:16:43,527 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #0
2021-02-24 12:16:43,527 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #1
2021-02-24 12:16:43,527 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #2
2021-02-24 12:16:43,527 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #3
2021-02-24 12:16:43,527 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #4
2021-02-24 12:16:43,527 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #5
2021-02-24 12:16:43,527 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #6
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #7
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #8
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #9
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #10
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #11
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #12
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #13
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #14
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #15
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #16
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #17
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #18
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #19
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #20
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #21
2021-02-24 12:16:43,528 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #22
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #23
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #24
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #25
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #26
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #27
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #28
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #29
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #30
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #31
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #32
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #33
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #34
2021-02-24 12:16:43,529 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #35
2021-02-24 12:16:43,530 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #36
2021-02-24 12:16:43,530 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #37
2021-02-24 12:16:43,530 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #0
2021-02-24 12:16:43,531 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #1
2021-02-24 12:16:43,531 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #2
2021-02-24 12:16:43,531 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #3
2021-02-24 12:16:43,532 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #4
2021-02-24 12:16:43,532 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #5
2021-02-24 12:16:43,533 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #6
2021-02-24 12:16:43,533 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #7
2021-02-24 12:16:43,534 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #8
2021-02-24 12:16:43,534 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #9
2021-02-24 12:16:43,534 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #10
2021-02-24 12:16:43,535 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #11
2021-02-24 12:16:43,535 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #12
2021-02-24 12:16:43,536 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #13
2021-02-24 12:16:43,536 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #14
2021-02-24 12:16:43,537 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #15
2021-02-24 12:16:43,537 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #16
2021-02-24 12:16:43,537 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #17
2021-02-24 12:16:43,538 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #18
2021-02-24 12:16:43,538 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #19
2021-02-24 12:16:43,539 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #20
2021-02-24 12:16:43,539 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #21
2021-02-24 12:16:43,539 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #22
2021-02-24 12:16:43,540 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #23
2021-02-24 12:16:43,540 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #24
2021-02-24 12:16:43,541 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #25
2021-02-24 12:16:43,541 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #26
2021-02-24 12:16:43,541 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #27
2021-02-24 12:16:43,542 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #28
2021-02-24 12:16:43,542 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #29
2021-02-24 12:16:43,542 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #30
2021-02-24 12:16:43,543 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #31
2021-02-24 12:16:43,543 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #32
2021-02-24 12:16:43,544 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #33
2021-02-24 12:16:43,544 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #34
2021-02-24 12:16:43,545 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #35
2021-02-24 12:16:43,545 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #36
2021-02-24 12:16:43,546 Dummy-42 SubJobXMLList ::_getItem :362 DEBUG : Requesting subjob: #37
2021-02-24 12:16:43,565 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 612
2021-02-24 12:16:43,595 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 613
2021-02-24 12:16:43,611 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 614
2021-02-24 12:16:43,625 Dummy-42 GangaRepositoryXML ::index_write :354 DEBUG : Writing index: 615
2021-02-24 12:16:43,657 GANGA_Update_Thread_JobRegistry_Monitor Registry ::keys :377 DEBUG : keys
2021-02-24 12:16:43,657 GANGA_Update_Thread_JobRegistry_Monitor Registry ::ids :359 DEBUG : ids
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__defaultActiveBackendsFunc:1027 DEBUG : Running over fixed_ids: [580, 581, 586, 587, 588, 589, 590, 593, 594, 606, 612, 613, 614, 615]
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,658 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Registry ::__getitem__ :320 DEBUG : __getitem__
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__defaultActiveBackendsFunc:1061 DEBUG : Returning active_backends: {"Dirac" : [589, 612, 613, 614, 615, ], }
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::_checkActiveBackends:1198 DEBUG : Active Backends: {"Dirac" : [589, 612, 613, 614, 615, ], }
2021-02-24 12:16:43,659 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :332 DEBUG : *: backend=Dirac, isOwner=False, joblist=[589, 612, 613, 614, 615], queue=3
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :336 DEBUG : Lock acquire successful. Updating jSet ['612', '589', '615', '614', '613'] with ['589', '612', '613', '614', '615'].
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :342 DEBUG : Dirac backend job set exists. Added ['589', '612', '613', '614', '615'] to it.
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::addEntry :353 DEBUG : **: backend=Dirac, isOwner=False, joblist=['589', '612', '613', '614', '615'], queue=3
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::_checkActiveBackends:1218 DEBUG : jList: ['589', '612', '613', '614', '615']
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :647 DEBUG :
Processing Function: diskSpaceCheckJobInsertor
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :655 DEBUG : cbHookEntry.enabled: False
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :656 DEBUG : (time.time() - cbHookEntry._lastRun): 1698.3125422000885
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :657 DEBUG : cbHookEntry.timeout: 30
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :647 DEBUG :
Processing Function: timeoutCheck
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :655 DEBUG : cbHookEntry.enabled: True
2021-02-24 12:16:43,660 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :656 DEBUG : (time.time() - cbHookEntry._lastRun): 2.1479156017303467
2021-02-24 12:16:43,661 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :657 DEBUG : cbHookEntry.timeout: 0
2021-02-24 12:16:43,661 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :660 DEBUG : Running monitoring callback hook function timeoutCheck(**{'thisDict': <GangaCore.Utility.threads.SynchronisedObject object at 0x7fe2243983c8>})
2021-02-24 12:16:43,661 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :676 DEBUG :
Running runClientCallbacks
2021-02-24 12:16:43,661 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::__monStep :679 DEBUG : Finished runClientCallbacks
2021-02-24 12:16:43,661 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :600 DEBUG : Finished Step
2021-02-24 12:16:43,661 GANGA_Update_Thread_JobRegistry_Monitor Local_GangaMC_Service::run :605 DEBUG : Wait Condition
2021-02-24 12:16:43,757 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:43,757 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:43,757 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,757 Dummy-42 Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:43,757 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,759 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:43,759 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:43,759 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,760 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: templates.metadata
2021-02-24 12:16:43,783 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: templates
2021-02-24 12:16:43,799 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:43,799 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:43,799 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,799 Dummy-42 Registry ::__len__ :336 DEBUG : __len__
2021-02-24 12:16:43,801 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:43,801 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:43,801 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,802 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: box.metadata
2021-02-24 12:16:43,824 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: box
2021-02-24 12:16:43,883 Dummy-42 Registry ::shutdown :659 DEBUG : Shutting Down Registry
2021-02-24 12:16:43,884 Dummy-42 Registry ::shutdown :660 DEBUG : shutdown
2021-02-24 12:16:43,884 Dummy-42 Registry ::values :385 DEBUG : values
2021-02-24 12:16:43,885 Dummy-42 GangaRepositoryXML ::shutdown :262 DEBUG : Shutting Down GangaRepositoryLocal: tasks
I hope this helps. Is there an option to start the log at a specific time and to keep everything? Edit: The warning is actually in my console scrollback somewhere, I will try to post a longer log, because I think all that output was after ^C
Here are some 46000 more lines (all during the exiting process), this is how far my tmux history goes back: console-log.txt
The very beginning, where I run queues was not captured, but I think there were two instances of the BackendMonitor running and a single subjob was finalising. Apparently, ganga keeps repeating some procedures
Closing this as the queues system for the monitoring is no longer used. It has been replaced with the asyncio improvements.