Open on google colab

[Marconefs]

Anyone knows if there is a way to open dataset files from the google storage(where the dataset is located) directly on google colab?

[amberbhardwaj]

Do you want to open from web or from this repo? Could you please elaborate a bit?

[mcgridles]

I am also interested in accessing the dataset files, I am trying to access them through colab using gcsfuse i.e. mount the storage bucket to the colab space:

!mkdir -p data/training
!gcsfuse waymo_open_dataset_v_1_2_0_individual_files/training/ data/training

It could be that I don't have the proper access to the Waymo buckets meaning this is probably not the correct way to do it so any help would be appreciated.

Edit: I figured it out, I needed to just mount the bucket instead of folders inside the bucket.

!mkdir -p data/
!gcsfuse waymo_open_dataset_v_1_2_0_individual_files/ data/

[peisun1115]

Nice! . I am glad that it worked

[JMarcan]

Hi @mcgridles, does it work for you? I tried the same, the colab mounts the folder but there are no data inside. Do I miss something?

!mkdir -p data/
!gcsfuse waymo_open_dataset_v_1_2_0_individual_files/ data/

Output:

Using mount point: /content/data
Opening GCS connection...
Opening bucket...
daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: setUpBucket: OpenBucket: Unknown bucket "waymo_open_dataset_v_1_2_0_individual_files/"

[mcgridles]

@JMarcan I thought it worked because I wasn't getting any errors but I also ended up with an empty folder like you mentioned. I ended up mounting each folder individually like this:

!mkdir -p data/training
!gcsfuse --only-dir training/ waymo_open_dataset_v_1_2_0_individual_files data/training/

!mkdir -p data/testing
!gcsfuse --only-dir testing/ waymo_open_dataset_v_1_2_0_individual_files data/testing/

Then you can find your training data in data/training and your testing data in data/testing.

[xpharry]

It was close. But the bucket name was incorrect and a tag before the bucket name was missing. The following command worked for me and gave the full access to the data.

!mkdir -p data/
!gcsfuse --implicit-dirs waymo_open_dataset_motion_v_1_1_0 data