Verify that MCC works with OPeNDAP URLs

[lewismc]

Let's write some more tests for the MCC which work with the following URLs

https://gpm1.gesdisc.eosdis.nasa.gov:443/opendap/GPM_L3/GPM_3IMERGM.03/2014/3B-MO.MS.MRG.3IMERG.20140312-S000000-E235959.03.V03D.HDF5

https://goldsmr4.gesdisc.eosdis.nasa.gov:443/opendap/MERRA2_MONTHLY/M2IMNXINT.5.12.4/1980/MERRA2_100.instM_2d_int_Nx.198001.nc4

http://opendap.jpl.nasa.gov:80/opendap/allData/smap/L3/RSS/V2/monthly/SCI/2015/RSS_smap_SSS_monthly_2015_04_v02.0.nc

[Omkar20895]

@lewismc To access some datasources we need to have authentication in the script and that is missing at present in our code.

[Omkar20895]

So It tried uploading the remote file directly from here and I got the error in the image below for all the three urls.

I think the podaac mcc API itself lacks the authentication service for some data sources but I am not sure about it. Thanks.

[lewismc]

Hi @Omkar20895 yes you are right it does. The correct way to address this is for us to implement the support over on the PO.DAAC side. I'll keep this issue open for a while. I wonder if we could provide a work around in Podaacpy e.g. check if it is a OPeNDAP URL and wrap the HTTP request using Pydap. WDYT?

[Omkar20895]

I don't know, I will work around with Pydap and then comment here.

[Omkar20895]

Ok, so I worked with pydap and thought I could download the remote file using pydap and then upload it using check_local_file() for mcc check instead of using check_remote_file(), but pydap client returns a pydap object. @lewismc

[lewismc]

@Omkar20895 looks like we may need to implement a NetCDFReponse

[Omkar20895]

WOW! I missed that in the docs! 😅 I will work on it. So the implementation would be downloading the file using pydap and then upload it using check_local_file() functionality. Any suggestions are welcome. Thanks.

[ffang10]

Most DAP tools (Panoply etc.) have this model of accessing .dds, .das (both metadata), and then .dods (binary data) responses. The netcdf Java or C libraries may support that. For compliance checkers only the first two (metadata) responses are relevant. The netcdf response in Pydap seems to be an add-on. The doc implies it works better for grid data, whereas we need a reliable compliance checker for swath and other processing-level data as well so it may have its limits.

Incidentally one type of OPeNDAP server, Hyrax, also implemented similar netcdf responses. The 3 URLs are all Hyrax servers, from which you can also download data in netcdf by appending .nc (or .nc4) and other constraint expressions to them.

[lewismc]

Thanks @ffang10

@Omkar20895, even if the NetCDF response returned by Pydap may work better on gridded data as oppose to swatch and other granularities, I still think that having functionality just as you've described will be of use. We can iterate on it. If you would like some assistance then let me know.

[Omkar20895]

@lewismc I will send a PR addressing this and as you have mentioned we will iterate on it with all the suggestions that we get. Thank you very much @ffang10

[Omkar20895]

There was an error that I was facing during installation of pydap with python3, I have opened an issue here, please take a look at it. @lewismc

[lewismc]

Thanks for looking into it @Omkar20895 I've replied over on the Pydap issue.

[Omkar20895]

OK, since I couldn't find any documentation on pydap.responses.netcdf module I went ahead and digged a little using this here. I have also opened the pydap.netcdf.responses package that is installed in my local machine(when I run pip), I had no clue how to use it. I think we need a lot of help from pydap here. 😓

@lewismc I would be glad if you could suggest something.

[lewismc]

Yes OK, lets address the issue over on Pydap https://github.com/pydap/pydap/issues/70 before we inherit her in Podaacpy.

[Omkar20895]

Yes, I have written some code but ended up with some errors, coming back to this after releasing a stable version of pydap is a good idea for now. Thanks :) @lewismc