Relative motion and impact to positioning over time

[ambarb]

I noticed some time ago that bluesky seems to perform relative motion with respect to the readback value. Almost all of the EPICS motor implementations i have seen for "jogging" or incremental steps perform the relative motion with respect to the setpoint. At first, I accepted this difference (bluesky not utilizing EPICS jog (PV_prefix.Mtr.TWF and PV_prefix.Mtr.TWR) or bluesky not acting the same as EPICS jog/tweak).

I speak about this in terms of plan_stubs like pbs.mvr and all of the standard "relative plans". tweak is also probably affected. Maybe I should have entered this question/concern in the ophyd api issues but I am not sure how isolated it is to ophyd.

Expected Behavior

After having used bluesky for awhile, the current approach to relative motion can cause headaches in consistency for the data and the experimental setup. We have encoded motors, and in a perfect world, the employed approach in bluesky should be exactly the same. However, this world is imperfect and we have to prioritize improvements on the beamline - often that means not fixing noisy encoders if the noise level is within the required tolerance.

Current Behavior

There are two cases we can demonstrate that relative motion to the "readback" position with noisy encoders that can cause a problem. In general, the first motions are "okay" but one can see that repeated relative motions, like relative scans, will likely cause the setpoint to drift from the ideal value or just cause confusion in book keeping.

Additionally, advance plotting an analysis based on motor positions becomes more challenging to automate because then you have to know the error bar for the analysis at each position. Many of our motions are not "linear" so the error bar is not constant either in terms of encoder readback noise or conversion to physical values.

1. We devoted a cumulative few months of study/work with 3 controls experts, the vendor's tech and engineer, an electrical engineer, and floor vibration measurements to identify noise. We were able to reduce noise to improve the performance so that working with the setpoint we could get reproducible motion control with an acceptable positional stability. However, the readback value still fluctuates quite a bit that we plot all data versus the setpoint. This is for the beamline energy - so you can imagE that the incident beamline energy, coupled with fixed HKL scanning, create a build up of error that becomes confusing and likely unacceptable.
2. For nanopositioning, we have a complex controller that handles the motion control. We have proved that using the setpoint in the analysis gives accurate and reliable results, where as the readback has an unacceptable level of noise. This readback fluctuation is something that we probably cannot expect the vendor to fix.
   • For one optic that has two scanning two axes, this is usually fine and maybe the user gets a little confused to where the optic is "supposed to be", especially if the optic allows for a large beam.
   • When we are using two optics (each with 2 independent axes), the relative alignment of theses optics to each other can diverge over time and adversely impact the data.

In these cases, confusion propagates because I don't know in looking though the data or electronic notes capture by others when I should trust the setpoint and when I should not - because the noise is so large on the readback that I cannot know when the readback is supposed to "match" the setpoint. In the end, starting from scratch in setting up may be the fastest thing to do, which is frustrating for the user and then there is more risk to the experiment.

Possible Solution

I can foresee four approaches, but I would be happy to discuss the problem further

• bluesky/ophyd utilizes EPICS TWR and TWF for relative motions for EpicsMotor device
• bluesky/ophyd creates its own equivalent TWR and TRF by moving relative to the setpoint
• Either of the above but with a specialized argument True/False argument in defining the EpicsMotor device

Steps to Reproduce (for bugs)

I think there is enough information here to understand the problem, but I am happy to provide information/demonstration if need be.

Context

See current behavior .

Your Environment

[ambarb]

It isn't clear from this initial report, but relative motion in plans like rel_scan() are also impacted by this. I have consistently used rel_scan in plans and have not had reproducible data (because of the technical issues outlined above for noisy motors and HKL).

Another impact to new users of bluesky is the inaccuracy of print_summary() applied to plans like rel_scan() because the user sees that nothing moves on the screen and then edit the plan arguments. When they pass this to the RE, then really crazy stuff happens. I've seen people do this and be very confused.

If all relative plans are adjusted so the trajectory is assembled in the generator prior to starting motion, then print_summary will make sense, check_limits becomes more useful, and data analysis is improved because the step size is constant.

Happy to provide examples if people want to see more.

[jklynch]

I do not foresee a time in the near future when I will be able to look at this. I recommend asking the AST.

[ambarb]

Thanks. I will just add it to the projects on Jira. I wasn't hassling you to do it and understand that everyone is busy.

[ZLLentz]

To fix this, we probably need to expand the bluesky device interface definitions (and make corresponding updates in ophyd). Looking at https://blueskyproject.io/bluesky/hardware.html#movable-or-settable-device, we don't have any "standard" way for bluesky to check a movable device's last set position. How to do this for a standard EPICS motor record is pretty clear, how to do it in general is not defined. The source code here for all the relative set handling relies on the special attribute we've defined to get the normal readback position (with a fallback of read-and-check-the-first-key): https://github.com/bluesky/bluesky/blob/b7d666e65cf4ef556fb46b744c33264c8e3f7507/bluesky/preprocessors.py#L988

I think we need to define an e.g. set_position attribute or some other reliable resource for bluesky to check that's available on all movable devices and redefine these relative plans using that resource instead of the position attribute.

[untzag]

I remember that @coretl had some language that we liked at one point. We were going to abandon the position attribute as ambiguous and introduce two new unambiguous attributes. I liked position and destination but there was another idea.

[coretl]

4th point of https://github.com/bluesky/bluesky/issues/1517 suggests we talked about setpoint (where we asked it to go) and position (where it actually is). I can't remember whether that was settled though...

[coretl]

Although I also found https://github.com/bluesky/bluesky/pull/1502#issuecomment-1111175618 which suggests setpoint and readback

[tacaswell]

My memory of early ophyd is that we thought the read back was more reliable than the set point (and definitely than a Python-side set point!). I think that @ambarb has made clear we were (I was) wrong about that.

I agree adding a new protocol for "this is where I should be" and "this is my One True Location" are. Maybe Locatable ?

I do not think we should require everything that is scanned to be a Locatable, but if we find one make use of it as a fast-path to improving operations.

[coretl]

I've made a new issue #1535 to track the creation of a protocol that would support this. Please put comments for the name of the protocol and its methods there