geos-chem-cloud
geos-chem-cloud copied to clipboard
geoschem
Syncing gcgrid S3 bucket with Dalhousie's FTP server?
Maggie Marvin at the University of Edinburgh asked whether the data on Dalhousie's FTP can be also uploaded to S3.
My questions are:
- Which part of Dalhousie's data are not on S3 right now?
My reference is http://wiki.seas.harvard.edu/geos-chem/index.php/Downloading_GEOS-Chem_source_code_and_data#Dalhousie_data_directory_archive
Looks like most of them are metfields and should largely duplicate s3://gcgrid/ ?
Further, custom nested fields can be cropped from the global data: s3://gcgrid/GEOS_0.25x0.3125/GEOS_FP/
- How large are those data? If it is small we can probably just upload them. If not then it involves talking to Amazon again.
- Who is managing Dalhousie's FTP server currently? Is it possible to coordinate?
Dal is storing the native-resolution GEOS-FP (0.25 x 0.3125) and MERRA-2 data (0.5 x 0.625) on the ComputeCanada server. I believe this is too large to store on Randall's server rain.ucis.dal.ca.
Jun Meng would be the best one to talk to re: Dal server support.
I'm happy to use s3://gcgrid/GEOS_0.25x0.3125/GEOS_FP/ and crop down to my domain (SE Asia), but I'm intending to run a nested simulation at 0.25x0.3125 resolution for 2017-present, and I was not able to find met fields for this resolution and timeframe on S3. However, it looks like met fields for these years might be available for some of the nested domains from the Dalhousie server at rain.ucis.dal.ca?
@MargaretMarvin as a temporary solution you can transfer those data from rain.ucis.dal.ca to EC2, and even upload them to your own S3 bucket for later use. Transferring data into AWS is free. The storage cost on S3 is $23/TB/month and I don't think nested metfields would exceed 1 TB.
I am interested in running a nested simulation at 0.25x0.3125 over China. I found that s3://gcgrid/GEOS_0.25x0.3125_CH/GEOS_FP/ only contains 2013 and 2016, whereas ftp://rain.ucis.dal.ca/ctm/GEOS_0.25x0.3125_CH.d/GEOS_FP/ contains met data from 2012 to now. Yes, it is not hard to transfer those data to EC2 or our own S3, but would be great if Amazon agree with you to upload those missing data.
