GCP GCS Notification Testing/Updates

[BryonLewis]

This is some testing of the GCS Notification endpoints. These endpoints are used to inform DIVE that new data has been uploaded to a GCP bucket and to process it properly.

The ultimate idea is to make it so that new videos or annotations can be added to a GCP bucket and are automatically imported and updated. The GCS Notifications send a rest request that will automatically indicate that new data has been sent. Then DIVE needs to process that and:

• Add new videos/images if they exist and conver them if they need to be converted by kicking off jobs
• If new annotations are added they need to be imported and associated with the proper video file
• Update any videos/image sequences that have changed.

TODO:

• [x] Test environment for testing GCP notifications
  • [x] MinIO service with data imported
  • [x] Script to add new data to the MinIO bucket
  • [x] Script to push notifications about the new data and inform DIVE to update
• [x] Support for Folders and Collections as mounting points
• [ ] Maybe a modification so that only new items in the ObjectId path are updated instead
  • [ ] Note: that params for assetStore importData could contain {'importPath': 'path'} and I think it would handle this properly.
• [ ] Only processing of newer data and don't try to conver older data
• [ ] Testing for adding new annotation file (CSV/JSON with corresponding video name)
• [ ] Testing for updating video with same name but new content
• [ ] Testing for Deletion of data from the S3 bucket using eventType: OBJECT_DELETE to delete data