spyglass icon indicating copy to clipboard operation
spyglass copied to clipboard

Export process with non-nwb external files

Open samuelbray32 opened this issue 5 months ago • 1 comments

Describe the bug

  • The docker export process uses Dandi and the AnalysisNwbfile table to log accessed files and share intermediate and final results

  • For historical and/or data type reasons, some tables store external files in a different format

    • ClusterlessDecodingV1
    • SortedSpikesDecodingV1
    • v0.SpikeSortingRecording
    • v0.SpikeSorting
    • v0.SpikeSorting()
    • v0.Waveforms()
    • v0.QualityMetrics()
  • These external files are not logged in the export process, nor are they shareable through Dandi

Solutions

  • Short term
    • If these are "intermediate" analysis files (not directly accessed in producing final results/figures) this shoudn't raise any errors when running in the Docker container.
    • File would not be available to someone trying to do new analysis from intermediate results in Docker container
  • Medium term
    • Recompute functionality (#917) would allow the Docker user to regenerate the missing files when needed
    • Bonus: we could potentially clean up these files on our server
  • Long term
    • Increase priority of storing files as nwb in new pipelines
    • Already done in spikesorting.v1 (Thanks @khl02007 )
    • Non-trivial for some filetypes. Can rely on re-compute in these cases (?)

samuelbray32 avatar Sep 04 '24 15:09 samuelbray32