Cannot download .swc files for axon_geometry experiments

[mlemyco]

Trying to run download_segments.py but it's not downloading the appropriate files: this loop and this loop do nothing

Successfully makes the directories but does not download anything.

[tathey1]

weird, it works for me.

If I remember correctly, you are on windows, while jacopo and i are on mac. Maybe it has something to do with paths. For example, the "prefix" variable might use backslashes for you, when it may be necessary to use forward slashes for bucket.objects.filter(Prefix=prefix) to work. Can you try hard coding the path "s3://open-neurodata/brainlit/axon_geometry/brain1/segments_swc/" and seeing what happens? Is this what your script uses for the prefix variable?

[JacopoTeneggi]

Hi! Thank you for reporting this issue.

It seems to be working on my laptop, so it could be an AWS initialization/authorization error.

Could you please provide the console output when you try and run download_segments.py? Do you see any warning messages regarding AWS/S3 initialization?

One other thing that you could try to do is add

 print(list(bucket.objects.filter(Prefix=prefix)))

before the for loop you mentioned. You should be getting an output similar to

[s3.ObjectSummary(bucket_name='open-neurodata', key='brainlit/axon_geometry/brain1/segments_swc/2018-08-01_G-002_consensus.swc'), s3.ObjectSummary(bucket_name='open-neurodata', key='brainlit/axon_geometry/brain1/segments_swc/2018-08-01_G-007_consensus.swc'), s3.ObjectSummary(bucket_name='open-neurodata', key='brainlit/axon_geometry/brain1/segments_swc/2018-08-01_G-011_consensus.swc'), s3.ObjectSummary(bucket_name='open-neurodata', key='brainlit/axon_geometry/brain1/segments_swc/2018-08-01_G-012_consensus.swc'), s3.ObjectSummary(bucket_name='open-neurodata', key='brainlit/axon_geometry/brain1/segments_swc/2018-08-01_G-013_consensus.swc'), s3.ObjectSummary(bucket_name='open-neurodata', key='brainlit/axon_geometry/brain1/segments_swc/2018-08-01_G-014_consensus.swc'), s3.ObjectSummary(bucket_name='open-neurodata', key='brainlit/axon_geometry/brain1/segments_swc/2018-08-01_G-020_consensus.swc'), ...

As Thommy pointed out, it is probably a Windows/Mac difference in how os.path.join concatenates paths.

[mlemyco]

Hi, thank you for the guidance!

This is the console output when I try to run download_segments.py with the print statement:

Downloading segments to C:\Users\mycol\OneDrive\Documents\GitHub\neurodata\brainlit\experiments\axon_geometry\data
[]
[]

No warnings--simply those 3 lines. The output is the same when I try hardcoding the path into the prefix variable.

This is the output when I add a couple more print statements to see what prefix and seg_dir are:

Downloading segments to C:\Users\mycol\OneDrive\Documents\GitHub\neurodata\brainlit\experiments\axon_geometry\data
prefix - brainlit\axon_geometry\brain1\segments_swc
seg_dir - C:\Users\mycol\OneDrive\Documents\GitHub\neurodata\brainlit\experiments\axon_geometry\data\brain1\segments_swc
[]
prefix - brainlit\axon_geometry\brain2\segments_swc
seg_dir - C:\Users\mycol\OneDrive\Documents\GitHub\neurodata\brainlit\experiments\axon_geometry\data\brain2\segments_swc
[]

Do you possibly see anything wrong?

[tathey1]

what about if you use forward slashes for prefix instead of backslashes?

i.e. prefix = "brainlit/axon_geometry/brain1/segments_swc"

[mlemyco]

Oof, I completely forgot to change the backslashes to forward slashes when I hardcoded the path in. That worked! It downloaded all the files now--thank you so much!

[tathey1]

@JacopoTeneggi Any reason why we need to use os.path to make the s3 path? Can prefix just be hardcorded?

[JacopoTeneggi]

Yes it definitely can, and should

[github-actions[bot]]

Stale issue message