OPeNDAP plug-in issues

[kyang2014]

By enabling opendap_model in the compass_viewer, I can build HDF-compass on Linux. So I can test the opendap plug-in at least from my development environment.

In general, One can open a URL served via DAP via "Open Resource". A simple netCDF style DAP service can be opened with the following limitations.

1. The current opendap model misses group attributes (at least for root)

The attributes under the root group(or global attributes) can not be displayed. It seems the feature is missing.

" File "/mnt/scr1/kent/hdf-compass-kent/hdf-compass/build/HDFCompass/out00-PYZ.pyz/compass_viewer.frame", line 388, in on_menu_reopen File "/mnt/scr1/kent/hdf-compass-kent/hdf-compass/build/HDFCompass/out00-PYZ.pyz/opendap_model", line 272, in init AttributeError: 'NoneType' object has no attribute 'attributes'

Note: Pydap client on the command-line can successfully retrieve the global attribute. So this is an issue related to the opendap model. Pydap output is as follows:

dataset = open_url('http://test.opendap.org/dap/data/nc/coads_climatology.nc') print dataset.attributes {'NC_GLOBAL': {'history': 'FERRET V4.30 (debug/no GUI) 15-Aug-96'}, 'DODS_EXTRA' : {'Unlimited_Dimension': 'TIME'}}

I think it is promising to fix this issue. It seems the opendap_model just forgets considering the global attribute case.

2. Can not serve any object names that contain special characters

The opendap plug-in can only address the object names that include underscore,letter and numeric values(CF naming conventions). So essentially all HDF5 data served by the default option of the Hyrax OPeNDAP services can not be served since a "/" is always included in an HDF5 object name. Symptom: the plug-in just opens the file without displaying any objects. The file name displayed by the plug-in looks like to be escaped with %...h5 for the dot in an HDF5 file name.

Pydap command-line client doesn't have an issue to open an HDF5 file served by the default HDF5 handler although I may see %.. displayed for the special characters inside the variable names.

3. Can and potentially will not serve compound datatype objects well because of the limitation of pydap

My version of the pydap(installed by using pip) can only handle a scalar compound datatype HDF5 dataset. So we should NOT expect the opendap module to be able to serve compound datatype objects well even if bugs are fixed.

[hyoklee]

@kyang2014, can you explain how to enable OPeNDAP support in the current compass? I tried

import opendap_model

under compass_viewer/init.py but nothing happens. Is there a GitHub branch that enabled OPeNDAP?

[kyang2014]

That's what I did and it worked. Have you "pyinstaller HDFcompass.spec" from the scratch?

[giumas]

I confirm @kyang2014 approach, I did the same. @hyoklee did you try python HDFCompass.py and then something like: http://test.opendap.org/opendap/hyrax/data/hdf5/grid_1_2d.h5 in Open resource file?

[hyoklee]

Thanks for reply. I thought it was not working because the URL from the latest Hyrax

https://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5

did not work.

However, I realized that the following URL works:

http://test.opendap.org/dap/data/nc/coads_climatology.nc

Why doesn't compass work with eosdap? Any clue?

[giumas]

@jreadey, this comment is not so clear for me: https://github.com/HDFGroup/hdf-compass/blob/master/compass_viewer/init.py#L160

Which issue arised? I have frozen it with pyinstaller without troubles.

[hyoklee]

@jreadey said enabling opendap messed up Windows packaging.

[kyang2014]

What's your definition of not working? I remembered that It works for me if you change http to https.

[hyoklee]

[giumas]

@jreadey said enabling opendap screwed Windows packaging.

Most likely, the work done for 3.0.0 of PyInstaller fixed the issue. Incidentally, this hack in https://github.com/HDFGroup/hdf-compass/blob/master/HDFCompass.spec#L29 is not more required.

[kyang2014]

weird, Which platform did you try?

[hyoklee]

iDAP Mac (Mavericks). I'll try Windows and Linux to see if the error is reproducible.

[kyang2014]

Try Linux one that we shared the same environment.

[giumas]

On Win7 64bit, only if added the s to http:

[jreadey]

I commented out the opendap plugin import in this commit: c5f075e05b4fa400ec21a3b3e4c05c9c6a63b267. Anyone who wants to experiement with it will need to uncomment that line!

[giumas]

@jreadey, do you remember the issue in the installer? I have not been able to replicate it!

[jreadey]

@giumas, no, I don't remember. I'll try it out once I have the builds working for Win & Linux.

[hyoklee]

Linux and Windows work file with eosdap.

[jreadey]

@kyang2014 - Did you try the running the built executable on another machine? I created this on Cent0S6: https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/linux-centos6_x86_64, but on running on another CentOS box, I got a undefined symbol error: xcb_wait_for_reply64 error on startup. Doing an install of libxcb.x86_64 resolved the problem.

So I guess pyinstaller isn't picking up all the needed libraries.

[hyoklee]

I can run it fine on Platypus.

See the attached screenshot. [image: Inline image 1]

HDF Product Designer: Ideate interoperable Bigdata for Citizen Science and IoT.

On Mon, Oct 5, 2015 at 11:51 PM, John Readey [email protected] wrote:

@kyang2014 https://github.com/kyang2014 - Did you try the running the built executable on another machine? I created this on Cent0S6: https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/linux-centos6_x86_64, but on running on another CentOS box, I got a undefined symbol error: xcb_wait_for_reply64 error on startup. Doing an install of libxcb.x86_64 resolved the problem.

So I guess pyinstaller isn't picking up all the needed libraries.

— Reply to this email directly or view it on GitHub https://github.com/HDFGroup/hdf-compass/issues/60#issuecomment-145741445 .

[hyoklee]

I learned that I can reach the following OPeNDAP server with Mac version of HDF Compass:

[1] http://giraffe:8080/opendap/data/hdf5/grid_1_2d.h5

Giraffe is the name used internally. Success with [1] indicates that The HDF Group's proxy server (Eosdap) doesn't work with Mac's PyDAP module. I think it's related to HTTPS redirection and specific to Mac platform only. Update user documentation as a limitation.

[jreadey]

@hyoklee: I'm able to open this URL: http://test.opendap.org/dap/data/nc/coads_climatology.nc. You can't open it on the HDF Group network?

[hyoklee]

Wren machine can't open Eosdap URL either but can open Giraffe URL. This behavior is consistent with my iDAP. I don't know what's so special about Mac's network library in handling response from proxy server.

[hyoklee]

This is not a good sign because the same https://eosdap URL can be opened by Java tools like Panoply without any problem. Python network stack (e.g., Pydap or urllib) or Mac's network library underneath is not working properly.

[jreadey]

@hyoklee - what's the eosdap URL that's not working? I'm not able to get a response from http://eosdap.hdfgroup.org:8080/opendap/hyrax/data/hdf5/grid_1_2d.h5 either from Compass or just doing wget on the command line.

[hyoklee]

Try: https://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5 Please note 's' next to http.

[jreadey]

Yes, that works for me. So doesn't that apply it's HDFGroup network specific?

[hyoklee]

Does the URL work in Seattle with HDFCompass? I think I gotta try it at home, later. It's so strange that https://eosdap... URL request works on web browsers like Safari but doesn't work for HDFCompass within The HDF Group network.

[jreadey]

I actually get an access denied error (403) using this url in Safari: https://eosdap.hdfgroup.org:8080/opendap/hyrax/data/hdf5/grid_1_2d.h5. (but it opens in Compass)

[hyoklee]

That's wrong URL, drop hyrax in URL. I updated my previous comment.

[jreadey]

Yes, that opens in Compass.