spyglass
spyglass copied to clipboard
Export process with non-nwb external files
Describe the bug
-
The docker export process uses Dandi and the
AnalysisNwbfile
table to log accessed files and share intermediate and final results -
For historical and/or data type reasons, some tables store external files in a different format
-
ClusterlessDecodingV1
-
SortedSpikesDecodingV1
-
v0.SpikeSortingRecording
-
v0.SpikeSorting
-
v0.SpikeSorting()
-
v0.Waveforms()
-
v0.QualityMetrics()
-
-
These external files are not logged in the export process, nor are they shareable through Dandi
Solutions
- Short term
- If these are "intermediate" analysis files (not directly accessed in producing final results/figures) this shoudn't raise any errors when running in the Docker container.
- File would not be available to someone trying to do new analysis from intermediate results in Docker container
- Medium term
- Recompute functionality (#917) would allow the Docker user to regenerate the missing files when needed
- Bonus: we could potentially clean up these files on our server
- Long term
- Increase priority of storing files as nwb in new pipelines
- Already done in
spikesorting.v1
(Thanks @khl02007 ) - Non-trivial for some filetypes. Can rely on re-compute in these cases (?)