I am trying to read a HDF file using the cis.read_data() function in Python. import cis import iris from cis import read_data from mpl_toolkits.basemap import Basemap import matplotlib.pyplot as plt import numpy as np a=cis.read_data('MOD06_L2.A2008001.0000.061.2017287164540.hdf','Cloud_Effective_Radius_16')

When I run this an error comes up:

ProductPluginException: An error occurred retrieving data using the product MODIS_L2. Check that this is the correct product plugin for your chosen data. Exception was TypeError: slice indices must be integers or None or have an __index__ method.

The same thing happens when I put in 'MODIS_L2' explicity as the plugin in the cis.read_data() function.

Am I using the wrong plugin? If not, then I can't work out how to read the file.

Aug 09 '19 16:08 tsarkany

Could you let me know which version of CIS and the other libraries you are using? Feel free to attach a print out of your environment.

Aug 09 '19 16:08 duncanwp

Dear Duncan, I am not sure whether this is what you are looking for but I have attached the output of the printenv command on Linux. When I type in cis version it outputs Using CIS version: 1.7.1 (Stable)

If this is not the right information then let me know! Best wishes, Tasmin

From: Duncan Watson-Parris [email protected] Sent: 09 August 2019 17:49:56 To: cedadev/cis [email protected] Cc: Tasmin Sarkany [email protected]; Author [email protected] Subject: Re: [cedadev/cis] Plugin not working for .hdf MODIS data (#22)

Could you let me know which version of CIS and the other libraries you are using? Feel free to attach a print out of your environment.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/cedadev/cis/issues/22?email_source=notifications&email_token=AM3S3X3NNSQK4W5A6QY37ODQDWN3JA5CNFSM4IKVVOEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD37GINA#issuecomment-519988276, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AM3S3X5FYNMZMRJNLJQRFRTQDWN3JANCNFSM4IKVVOEA.

Aug 09 '19 17:08 tsarkany

Printout environment.pdf

Aug 09 '19 17:08 tsarkany

OK, that's a nice recent version of CIS. To get the versions of the other Python packages could you send the output of running the command: conda list. Thanks!

Aug 10 '19 13:08 duncanwp

Created a new python 3 environment with cis to test this, getting the same error. The problem is in MODIS_L2.__field_interpolate on line 247 of cis/data_io/products/MODIS.py. I believe that all the slice indices now need to be specifically int type due to changes in numpy. The full error output is as follows:

Traceback (most recent call last): File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/AProduct.py", line 197, in get_data data = product_cls().create_data_object(filenames, variable) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/MODIS.py", line 299, in create_data_object coords = self._create_coord_list(filenames, variable) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/MODIS.py", line 269, in _create_coord_list lat_data = self.__field_interpolate(sd_lat) if apply_interpolation else sd_lat File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/MODIS.py", line 247, in __field_interpolate float(factor)) + output[int(factor / 2):(-1 * factor):factor, :] TypeError: slice indices must be integers or None or have an index method

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/init.py", line 41, in read_data data_list = read_data_list(filenames, variable, product) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/init.py", line 70, in read_data_list return DataReader().read_data_list(file_set, variables, product, aliases) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/data_reader.py", line 90, in read_data_list var_data = self._get_data_func(filenames, variable, product) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/AProduct.py", line 203, in get_data % (product_cls.name, type(e).name, e.args[0]), e) cis.data_io.products.AProduct.ProductPluginException: An error occurred retrieving data using the product MODIS_L2. Check that this is the correct product plugin for your chosen data. Exception was TypeError: slice indices must be integers or None or have an index method.

Aug 10 '19 14:08 w-k-jones

Ah yeah, more specifically it's because this value in the slicing [(-1 * factor / 2 + 1)] isn't converted to int type

Aug 10 '19 14:08 w-k-jones

This is the output of conda list

packages in environment at /home/linc4371/miniconda3:

Aug 12 '19 10:08 tsarkany

It looks like the easiest fix is to replace float division / with integer division // in those lines. Any obvious reason why that wouldn't work?

Aug 12 '19 10:08 adamcpovey

That floors the number, doesn't it? If so, would that break compatibility with the previous functionality? i.e: If (-1 * factor / 2 + 1) = 0.5 then was the 0th or 1st element of the array selected? Doing (-1 * factor // 2 + 1) will always select the lower element.

Aug 13 '19 11:08 simonrp84

numpy would previously select index 0 for 0.5. The only problem is trying to find out if the value was floored or cast as int when indexing, as that changes the behaviour for negative values

Aug 13 '19 11:08 w-k-jones

Trying it in my own code doesn't work as you end up with a 2030x1350 coordinate and a 2030x1354 data field. It appears some rounding was being done somewhere.

Aug 13 '19 11:08 adamcpovey

To be honest, I'm keen to remove this code entirely since there are a few nasty cases which I'm not convinced it's properly dealing with, this issue of the 4 'missing' coordinates being one of them.

This is a first go at making this work with the python-geotiepoints package, which is probably the best way to tackle this: https://github.com/duncanwp/cis_plugins/blob/master/modis_fixed.py

The MOD06 and MOD04 products seem to have slightly different structures though so it still isn't working as generally as I'd like.

Aug 13 '19 11:08 duncanwp

cis
cis copied to clipboard

Plugin not working for .hdf MODIS data

packages in environment at /home/linc4371/miniconda3:

Name Version Build Channel

cis cis copied to clipboard

Plugin not working for .hdf MODIS data

packages in environment at /home/linc4371/miniconda3:

Name Version Build Channel

cis
cis copied to clipboard