bluesky
scans crash(and bluesky becomes unusable) on a "a set() call is already in progress"
We have several scan suspenders that call for a shutter to be closed when beam becomes unavailable. Occasionally, when two or more of the suspenders have been triggered, an unhandled exception--the "set() call is already is progress" error alluded to in the title--is raised, apparently on an attempt to re-open the shutter when conditions return to normal. That set() call is never cleared, forcing an exit and restart of Bluesky. This does not happen every time that two suspenders are triggered. The suspenders have not been changed in more than four months, so this issue appears to be new with the most current release of Bluesky/Ophyd. This is problem for stable operation and has already happened twice in the last week.
set_and_wait as of #367 allows an infinite timeout (going against the original intention of the function). set getting held up there is probably just a symptom of your real problem though (race condition?).
This may also be related to https://github.com/NSLS-II/ophyd/pull/372 which was fixed at SRX via a hot-fix last cycle.
Where are the post-plans you are using?
def shuttergenerator(shutter, value):
return (yield from bp.rewindable_wrapper(bp.mv(shutter, value), False))
#ring current suspender
susp_rc = SuspendFloor(ring_current, 140, resume_thresh=140, sleep=10*60,
pre_plan=list(shuttergenerator(shut_b, 'Close')),
post_plan=list(shuttergenerator(shut_b, 'Open'))
)
When you say 'two or more' suspenders do you mean more than once in the same call to RE or more than one suspender has failed at the same time?
