mcap
mcap copied to clipboard
foxglove
mcap cli: Add support for Azure Storage Accounts and AWS S3 remote files
Public-Facing Changes
You can now use mcap cli with Azure Storage Accounts and AWS S3 remote files, as well as GCS.
Description
Uses gocloud.dev from Google to build generic clients for Azure, AWS and Google Cloud.
Note: I am yet to test all these changes on every cloud environment
- [ ] Verify on Azure
- [x] Verify on AWS S3
- [ ] Verify on GCS
Azure Usage
TODO
AWS Usage
- You must have an authenticated AWS session in your environment.
- You must set the aws region of the s3 bucket, either through
~/.aws/configorAWS_REGION=...
GSC Usage
Hopefully unchanged (need to check)
Hey @RileyEv, thanks! This will be useful.
I can confirm it works on GCS:
[~/work/mcap/go/cli/mcap] ((HEAD detached at riley/mcap-cli-s3-remote-file)) $ mcap info gs://dp-byoc-inbox-wyatt/demo.mcap
library: mcap go v1.0.1
profile: ros1
messages: 1606
duration: 7.780758504s
start: 2017-03-21T19:26:20.103843113-07:00 (1490149580.103843113)
end: 2017-03-21T19:26:27.884601617-07:00 (1490149587.884601617)
compression:
zstd: [14/14 chunks] [119.10 MiB/58.73 MiB (50.69%)] [7.55 MiB/sec]
channels:
(0) /diagnostics 52 msgs (6.68 Hz) : diagnostic_msgs/DiagnosticArray [ros1msg]
(1) /image_color/compressed 234 msgs (30.07 Hz) : sensor_msgs/CompressedImage [ros1msg]
(2) /tf 774 msgs (99.48 Hz) : tf2_msgs/TFMessage [ros1msg]
(3) /radar/points 156 msgs (20.05 Hz) : sensor_msgs/PointCloud2 [ros1msg]
(4) /radar/range 156 msgs (20.05 Hz) : sensor_msgs/Range [ros1msg]
(5) /radar/tracks 156 msgs (20.05 Hz) : radar_driver/RadarTracks [ros1msg]
(6) /velodyne_points 78 msgs (10.02 Hz) : sensor_msgs/PointCloud2 [ros1msg]
attachments: 0
metadata: 0
On Azure I'm not sure I'm getting the call right but I see this:
[~/work/mcap/go/cli/mcap] ((HEAD detached at riley/mcap-cli-s3-remote-file)) $ mcap info azblob://fgdemostorage.blob.core.windows.net/lake/gps__fix_7f6e605ad1e52d05162190ff17be80b6/device_id=dev_BDIlonk6XMUtsPUJ/date=20210322/8568660e2377d5621c17da43217c39bf
Failed to read file azblob://fgdemostorage.blob.core.windows.net/lake/gps__fix_7f6e605ad1e52d05162190ff17be80b6/device_id=dev_BDIlonk6XMUtsPUJ/date=20210322/8568660e2377d5621c17da43217c39bf: azureblob: Options.AccountName is required
I am not sure how it wants me to give it an account name. I do have an authenticated azure CLI session so my expectation would be I need no other config/params to do this.
on AWS I see this - the object definitely does exist:
[~/work/mcap/go/cli/mcap] ((HEAD detached at riley/mcap-cli-s3-remote-file)) $ mcap info s3://foxglove-wyatt-test-bucket/gps.mcap
Failed to read file s3://foxglove-wyatt-test-bucket/gps.mcap: failed to build read seek closer: blob (key "gps.mcap") (code=NotFound): operation error S3: GetObject, https response error StatusCode: 301, RequestID: NV48YNWKN5EDMWHN, HostID: xJyRtsJqDtovu+5FKWpQfctSuVFRHST+JS3sFSMyVG/aTUVeL/QZJo3AA5Hk1uUXj4H1YElcxrc=, api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
is that the call you'd expect to work?
edit:
[~/work/mcap/go/cli/mcap] (main) $ aws s3 ls s3://foxglove-wyatt-test-bucket/gps.mcap
2023-08-29 18:07:28 1454223 gps.mcap
aws
on AWS I see this - the object definitely does exist:
I think that means that something is not quite right with the authentication. I get the same message when trying to use your bucket which I expect I wouldn't have access to.
mcap/go/cli/mcap on mcap-cli-s3-remote-file [?] via 🐹 v1.21.0 took 2s
➜ go run . info s3://foxglove-wyatt-test-bucket/gps.mcap
Failed to read file s3://foxglove-wyatt-test-bucket/gps.mcap: failed to build read seek closer: blob (key "gps.mcap") (code=NotFound): operation error S3: GetObject, https response error StatusCode: 301, RequestID: 8508VF35EXXJ6PXC, HostID: EqRD7aMLM5DFPJv77sDPoPIO9AtjRnCWUXj89dZb1NgcTtG1nNM+REKIHdhdGkbSRZJgyJxd5FQ=, api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
exit status 1
How have you setup credentials? I've tested it with credentials from IAM Identity Center (SSO) and another auth method using aws-okta and both work for me.
mcap/go/cli/mcap on mcap-cli-s3-remote-file [?] via 🐹 v1.21.0
➜ aws s3 ls s3://mcap-testing/
2023-08-30 21:03:04 512262937 nuscenes.mcap
mcap/go/cli/mcap on mcap-cli-s3-remote-file [?] via 🐹 v1.21.0
➜ go run . info s3://mcap-testing/nuscenes.mcap
library: nuscenes2mcap
profile:
messages: 33886
duration: 19.192673s
start: 2018-07-24T04:28:47.604844+01:00 (1532402927.604844000)
end: 2018-07-24T04:29:06.797517+01:00 (1532402946.797517000)
compression:
lz4: [638/638 chunks] [741.37 MiB/487.76 MiB (34.21%)] [25.41 MiB/sec]
channels:
(1) /imu 1898 msgs (98.89 Hz) : IMU [jsonschema]
(2) /odom 937 msgs (48.82 Hz) : Pose [jsonschema]
(3) /map 1 msgs (0.05 Hz) : foxglove.Grid [protobuf]
(4) /semantic_map 1 msgs (0.05 Hz) : foxglove.SceneUpdate [protobuf]
(5) /tf 3002 msgs (156.41 Hz) : foxglove.FrameTransform [protobuf]
(6) /drivable_area 39 msgs (2.03 Hz) : foxglove.Grid [protobuf]
(7) /RADAR_FRONT 247 msgs (12.87 Hz) : foxglove.PointCloud [protobuf]
(8) /RADAR_FRONT_LEFT 262 msgs (13.65 Hz) : foxglove.PointCloud [protobuf]
(9) /RADAR_FRONT_RIGHT 258 msgs (13.44 Hz) : foxglove.PointCloud [protobuf]
(10) /RADAR_BACK_LEFT 253 msgs (13.18 Hz) : foxglove.PointCloud [protobuf]
(11) /RADAR_BACK_RIGHT 243 msgs (12.66 Hz) : foxglove.PointCloud [protobuf]
(12) /LIDAR_TOP 382 msgs (19.90 Hz) : foxglove.PointCloud [protobuf]
(13) /CAM_FRONT/image_rect_compressed 224 msgs (11.67 Hz) : foxglove.CompressedImage [protobuf]
(14) /CAM_FRONT/camera_info 224 msgs (11.67 Hz) : foxglove.CameraCalibration [protobuf]
(15) /CAM_FRONT/lidar 224 msgs (11.67 Hz) : foxglove.ImageAnnotations [protobuf]
(16) /CAM_FRONT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(17) /CAM_FRONT_RIGHT/image_rect_compressed 217 msgs (11.31 Hz) : foxglove.CompressedImage [protobuf]
(18) /CAM_FRONT_RIGHT/camera_info 217 msgs (11.31 Hz) : foxglove.CameraCalibration [protobuf]
(19) /CAM_FRONT_RIGHT/lidar 217 msgs (11.31 Hz) : foxglove.ImageAnnotations [protobuf]
(20) /CAM_FRONT_RIGHT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(21) /CAM_BACK_RIGHT/image_rect_compressed 218 msgs (11.36 Hz) : foxglove.CompressedImage [protobuf]
(22) /CAM_BACK_RIGHT/camera_info 218 msgs (11.36 Hz) : foxglove.CameraCalibration [protobuf]
(23) /CAM_BACK_RIGHT/lidar 218 msgs (11.36 Hz) : foxglove.ImageAnnotations [protobuf]
(24) /CAM_BACK_RIGHT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(25) /CAM_BACK/image_rect_compressed 216 msgs (11.25 Hz) : foxglove.CompressedImage [protobuf]
(26) /CAM_BACK/camera_info 216 msgs (11.25 Hz) : foxglove.CameraCalibration [protobuf]
(27) /CAM_BACK/lidar 216 msgs (11.25 Hz) : foxglove.ImageAnnotations [protobuf]
(28) /CAM_BACK/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(29) /CAM_BACK_LEFT/image_rect_compressed 219 msgs (11.41 Hz) : foxglove.CompressedImage [protobuf]
(30) /CAM_BACK_LEFT/camera_info 219 msgs (11.41 Hz) : foxglove.CameraCalibration [protobuf]
(31) /CAM_BACK_LEFT/lidar 219 msgs (11.41 Hz) : foxglove.ImageAnnotations [protobuf]
(32) /CAM_BACK_LEFT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(33) /CAM_FRONT_LEFT/image_rect_compressed 224 msgs (11.67 Hz) : foxglove.CompressedImage [protobuf]
(34) /CAM_FRONT_LEFT/camera_info 224 msgs (11.67 Hz) : foxglove.CameraCalibration [protobuf]
(35) /CAM_FRONT_LEFT/lidar 224 msgs (11.67 Hz) : foxglove.ImageAnnotations [protobuf]
(36) /CAM_FRONT_LEFT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(37) /pose 39 msgs (2.03 Hz) : foxglove.PoseInFrame [protobuf]
(38) /gps 39 msgs (2.03 Hz) : foxglove.LocationFix [protobuf]
(39) /markers/annotations 39 msgs (2.03 Hz) : foxglove.SceneUpdate [protobuf]
(40) /markers/car 39 msgs (2.03 Hz) : foxglove.SceneUpdate [protobuf]
(41) /diagnostics 22019 msgs (1147.26 Hz) : diagnostic_msgs/DiagnosticArray [ros1msg]
attachments: 0
metadata: 1
azure
For Azure I think the default format expected by the library is a little weird. It requires that the storage account be set as an env var. It can also be passed as an option when building the url representing the blob. I think your usage shows that the need for an env var and unusual format is probably going to be confusing for users and it'd be much more helpful if you can paste in a blob store link like https://mcaptesting.blob.core.windows.net/mcap/nuscenes.mcap (maybe prefixed with azblob:// instead of https://?) and then its able to put it into the format that the library would like.
I did manage to get it working though, by setting the env vars.
mcap/go/cli/mcap on mcap-cli-s3-remote-file [?] via 🐹 v1.21.0 took 2s
➜ az storage blob list --account-name mcaptesting --container-name mcap -o table
Name Blob Type Blob Tier Length Content Type Last Modified Snapshot
------------- ----------- ----------- --------- ------------------------ ------------------------- ----------
nuscenes.mcap BlockBlob Hot 512262937 application/octet-stream 2023-08-30T20:43:30+00:00
mcap/go/cli/mcap on mcap-cli-s3-remote-file [?] via 🐹 v1.21.0
➜ AZURE_STORAGE_ACCOUNT=mcaptesting AZURE_STORAGE_KEY=… go run . info azblob://mcap/nuscenes.mcap
library: nuscenes2mcap
profile:
messages: 33886
duration: 19.192673s
start: 2018-07-24T04:28:47.604844+01:00 (1532402927.604844000)
end: 2018-07-24T04:29:06.797517+01:00 (1532402946.797517000)
compression:
lz4: [638/638 chunks] [741.37 MiB/487.76 MiB (34.21%)] [25.41 MiB/sec]
channels:
(1) /imu 1898 msgs (98.89 Hz) : IMU [jsonschema]
(2) /odom 937 msgs (48.82 Hz) : Pose [jsonschema]
(3) /map 1 msgs (0.05 Hz) : foxglove.Grid [protobuf]
(4) /semantic_map 1 msgs (0.05 Hz) : foxglove.SceneUpdate [protobuf]
(5) /tf 3002 msgs (156.41 Hz) : foxglove.FrameTransform [protobuf]
(6) /drivable_area 39 msgs (2.03 Hz) : foxglove.Grid [protobuf]
(7) /RADAR_FRONT 247 msgs (12.87 Hz) : foxglove.PointCloud [protobuf]
(8) /RADAR_FRONT_LEFT 262 msgs (13.65 Hz) : foxglove.PointCloud [protobuf]
(9) /RADAR_FRONT_RIGHT 258 msgs (13.44 Hz) : foxglove.PointCloud [protobuf]
(10) /RADAR_BACK_LEFT 253 msgs (13.18 Hz) : foxglove.PointCloud [protobuf]
(11) /RADAR_BACK_RIGHT 243 msgs (12.66 Hz) : foxglove.PointCloud [protobuf]
(12) /LIDAR_TOP 382 msgs (19.90 Hz) : foxglove.PointCloud [protobuf]
(13) /CAM_FRONT/image_rect_compressed 224 msgs (11.67 Hz) : foxglove.CompressedImage [protobuf]
(14) /CAM_FRONT/camera_info 224 msgs (11.67 Hz) : foxglove.CameraCalibration [protobuf]
(15) /CAM_FRONT/lidar 224 msgs (11.67 Hz) : foxglove.ImageAnnotations [protobuf]
(16) /CAM_FRONT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(17) /CAM_FRONT_RIGHT/image_rect_compressed 217 msgs (11.31 Hz) : foxglove.CompressedImage [protobuf]
(18) /CAM_FRONT_RIGHT/camera_info 217 msgs (11.31 Hz) : foxglove.CameraCalibration [protobuf]
(19) /CAM_FRONT_RIGHT/lidar 217 msgs (11.31 Hz) : foxglove.ImageAnnotations [protobuf]
(20) /CAM_FRONT_RIGHT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(21) /CAM_BACK_RIGHT/image_rect_compressed 218 msgs (11.36 Hz) : foxglove.CompressedImage [protobuf]
(22) /CAM_BACK_RIGHT/camera_info 218 msgs (11.36 Hz) : foxglove.CameraCalibration [protobuf]
(23) /CAM_BACK_RIGHT/lidar 218 msgs (11.36 Hz) : foxglove.ImageAnnotations [protobuf]
(24) /CAM_BACK_RIGHT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(25) /CAM_BACK/image_rect_compressed 216 msgs (11.25 Hz) : foxglove.CompressedImage [protobuf]
(26) /CAM_BACK/camera_info 216 msgs (11.25 Hz) : foxglove.CameraCalibration [protobuf]
(27) /CAM_BACK/lidar 216 msgs (11.25 Hz) : foxglove.ImageAnnotations [protobuf]
(28) /CAM_BACK/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(29) /CAM_BACK_LEFT/image_rect_compressed 219 msgs (11.41 Hz) : foxglove.CompressedImage [protobuf]
(30) /CAM_BACK_LEFT/camera_info 219 msgs (11.41 Hz) : foxglove.CameraCalibration [protobuf]
(31) /CAM_BACK_LEFT/lidar 219 msgs (11.41 Hz) : foxglove.ImageAnnotations [protobuf]
(32) /CAM_BACK_LEFT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(33) /CAM_FRONT_LEFT/image_rect_compressed 224 msgs (11.67 Hz) : foxglove.CompressedImage [protobuf]
(34) /CAM_FRONT_LEFT/camera_info 224 msgs (11.67 Hz) : foxglove.CameraCalibration [protobuf]
(35) /CAM_FRONT_LEFT/lidar 224 msgs (11.67 Hz) : foxglove.ImageAnnotations [protobuf]
(36) /CAM_FRONT_LEFT/annotations 39 msgs (2.03 Hz) : foxglove.ImageAnnotations [protobuf]
(37) /pose 39 msgs (2.03 Hz) : foxglove.PoseInFrame [protobuf]
(38) /gps 39 msgs (2.03 Hz) : foxglove.LocationFix [protobuf]
(39) /markers/annotations 39 msgs (2.03 Hz) : foxglove.SceneUpdate [protobuf]
(40) /markers/car 39 msgs (2.03 Hz) : foxglove.SceneUpdate [protobuf]
(41) /diagnostics 22019 msgs (1147.26 Hz) : diagnostic_msgs/DiagnosticArray [ros1msg]
attachments: 0
metadata: 1
I'm a bit meh on the CLI tool working directly on remote files but I see we already have GCS support. Is there anything else you need in this PR for the AWS support? I don't see why we need to block this on adding Azure support - someone else can come along and add that later.
Is this PR ready for review? It is still marked as draft. If you have lost interest can we close it?
I'm gonna close this since there's been no response to my last inquiry for some time. If you decide to re-visit we can consider it again.