Make filename-setting switchable?

[danielballan]

Currently, APS is not generating Resource or Datum documents. The artifacts produced by detectors are not tracked by databroker, and therefore they are not accessible to scripts or GUIs (Xi-cam, bluesky-browser) that rely on databroker.

At NSLS-II, the DAMA group is able to take the position, "Artifacts produced by detectors will be treated as internal resources, and will not have metadata in filenames." While at least one beamline (LIX) has worked around ophyd to intervene in file naming and at least one other (CHX) routinely accesses files directly without renaming them, we have for the most part managed to hold the line on this, and we plan to continue to do so.

At APS, there are existing workflows that expect these artifacts. DataBroker is not the only consumer and, as the Bluesky Project is new middleware in this system, it must adapt itself to existing protocols. How can we do this?

Possible Approaches

1. Break the world! Update all file-based workflows at once to use databroker instead.
2. Modify the classes in ophyd.areadetector.filestore_mixins to support a mode where the Resource passively records the name set by the user / Area Detector rather than actively setting the name to a UUID4 during staging.
3. Write the file to an internal (UUID4-named) location but, during staging, create a soft-link or a full copy at the user-specified path.
4. The inverse of (3): write the file the user-specific path but create a soft-link or a full copy at an internal location.

From conversations with @prjemian, (1) is a nonstarter. For (3) and (4), doing full copies is a hard sell because while storage itself is cheap enough, the I/O of creating the copy would impose a burden on a resource in high demand and delay the next acquisition. But I don't think we discussed the possibility of using soft-links. Thoughts, @prjemian?

Another factor to keep in mind here is that, at APS, the files are typically promptly moved to a new location that is not accessible on the beamline network, so Resources would have to propagate with them and be updated or root_map-ed.

[prjemian]

soft links might be acceptable

[danielballan]

Notes from a Pilot call:

Paraphrasing @tacaswell from memory

Links are fragile. If some facilities need to let users customize the file names, ophyd should add a hook to enable that. Enforcing unique names is a matter for facility policy, not the library.

So let's do (2).

[stuartcampbell]

Would a 'suitcase-symboliclink' be something we would consider ?

[danielballan]

I think there is a fair worry there that links are fragile. The underlying files may be moved, and Data Broker can manage that seamlessly (root_map) but the links will break. Perhaps better to choose between exporting a materialized copy of the data or else using ophyd to customize the file name as desired.

[prjemian]

The concern is that links may be broken past the point of repair by a redefinition of root_map. As a spitball idea, a double-linked list comes to mind wherein a reference back to the datum and/or run is deposited into the external file. This is used to recover in the extreme event that an external file must be reunited with the databroker and all the usual means have failed.