brainlit icon indicating copy to clipboard operation
brainlit copied to clipboard

Published 20 hours ago • verified publisher icon neurodata

Add data validation script to utils

Open bvarjavand opened this issue 4 years ago • 2 comments

This would be a script that gets run on the cis server.

  • [ ] pulls random chunks of data from source octree /cis/local/jacs/...
  • [ ] pulls same chunks from cloud upload
  • [ ] verified chunks are identical

It would be a great QA script to instill confidence that uploads are valid.

bvarjavand avatar Aug 07 '20 18:08 bvarjavand

alternatively or additionally, use the get_data_ranges method here to enumerate the ranges for each mip and confirm that a file exists with the name of each possible range. May need to use the aws cli here for the bucket at s3://mouse-light-viz/precomputed_volumes/brain1

bvarjavand avatar Aug 08 '20 01:08 bvarjavand

It seems to make the most sense for there to be a utils/validation.py file which executes methods to validate already-uploaded CloudVolumePrecomputed volumes.

It would be able to be tested alongside the utils/upload.py tests which create a local CloudVolumePrecomputed volume at multiple (2) resolutions.

bvarjavand avatar Aug 08 '20 05:08 bvarjavand