Cannot load variable through s3fs but can brown on AWS S3 Explorer

[diptiSH]

trafficstars

Hello,

I am a new user of AWS datastore. I want to get 'sftlf' variable on AWS I am using the following code -

Connect to AWS S3 storage

fs = s3fs.S3FileSystem(anon=True) df = pd.read_csv("https://cmip6-pds.s3.amazonaws.com/pangeo-cmip6.csv") qstring="activity_id=='CMIP' & institution_id =='NCC' & source_id=='NorESM2-LM' & table_id=='fx' & experiment_id=='historical' & member_id=='r1i1p1f1' & variable_id=='sftlf'" df.query(qstring)

This returns the empty string.

I am not getting it through s3fs but I can browse it on AWS S3 Explorer. https://cmip6-pds.s3.amazonaws.com/index.html#CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/fx/sftlf/gn/v20190815/sftlf/

[rabernat]

Thanks for your question.

This probably means that the data have been retracted. They may still be in s3 but are not exposed by the catalog. cc @jbusecke.

[jbusecke]

AFAIK there have been no retractions on AWS so far (at least I havent done any).

But @rabernat advice is generally right. @diptiSH we usually do not delete stores, just unlink them from the catalog. This is done so that retracted or otherwise 'wonky' datasets can still be used by users who specifically want this. May I ask if there was a specific page that recommended the AWS explorer route? We might want to add a warning there then.