spyglass icon indicating copy to clipboard operation
spyglass copied to clipboard

Populate should leave evidence of success state

Open CBroz1 opened this issue 1 year ago • 4 comments

Problem: It is possible to run common_behav.StateScript.populate for a given key and have no resulting insert. This leaves previously attempted keys in the key_source, which will be re-attempted if re-run. The current structure doesn't distinguish between 'attempted' and 'not attempted'.

Proposed: By modifying the secondary key to be nullable, we can retain a record of which keys have been attempted. Optionally, another secondary key could be added to be explicit (e.g., has_assoc_file: bool). I have seen this pattern on other tables. I propose migrating all populates to retain records of attempted keys with null entries

Considerations: In this case the cost of rerunning is small, a couple conditionals, but this habit opens the door to unnecessary processing time elsewhere. Stylistically, lab members tend to populate a specific key as needed, which will not proc this issue, but datajoint documentation often uses bare populates, so use habits may change as more folks try out spyglass

CBroz1 avatar Feb 26 '24 16:02 CBroz1