ccdproc

ccdproc copied to clipboard

Hi,

i was wondering if there are any plans to have Combiner take in a DQ style array for masking, (i.e. scaling to zero). I've been getting a lot of requests for a mscombine replacement, and I think people use this feature a lot. If there are no plans for this, I'm thinking about adding a wrapper around Combine to enable this, which would possibly end up in the one of the STScI packages... thoughts?

@SaraOgaz What exactly doesn't work for "DQ style mask arrays"?

ccdproc currently automatically casts the mask to a boolean array. Meaning that any "mask" value other than zero (so if it has any DQ flag) is treated as "masked" and ignored during combination:

>>> import ccdproc
>>> import numpy as np

>>> # image with "some" quality flags (totally flagged)
>>> ccd = ccdproc.CCDData(np.ones((3,3)), mask=np.ones((3,3), dtype=int), unit='adu')
>>> ccd.mask[1,1] = False  # let's unmask one pixel
>>> # image with no quality flag
>>> ccd2 = ccdproc.CCDData(np.ones((3,3)) * 2, mask=np.zeros((3,3), dtype=int), unit='adu')

>>> ccdcomb = ccdproc.Combiner([ccd, ccd, ccd2])
>>> ccdcomb.average_combine()
CCDData([[ 2.        ,  2.        ,  2.        ],
         [ 2.        ,  1.33333333,  2.        ],
         [ 2.        ,  2.        ,  2.        ]])

But I'm not sure if I understood the question correctly. So please correct me if I'm wrong. Or is it about the combine function rather than the Combiner class?

There is no objection in principle to supporting that; a little more detail and/or an example would be helpful.

Also, you may want to take a quick look at the Astropy Proposal for Enhancement for NDData, which add supporting for DQ planes at a more fundamental level. Even a simple "+1" or "-1" for that feature would be useful :)

@MSeifert04, I'm thinking of the situation where, in ccdproc language (and hopefully I'm understanding this package correctly), I have a flag array in my CCDData object, and I want to use a set of flag values in that flag array (in HST data's case, the flags are bitflags, so easy to test for inclusion of a certain flag in each pixel) to be used for masking when running Combiner. This is the equivalent to the 'dqbits' parametere in mscombine: http://stsdas.stsci.edu/cgi-bin/gethelp.cgi?mscombine.

I'm not sure how many telescopes outside of HST use this kind of DQ/flag arrays, so I'm not sure if this would only really be useful for HST data.

@mwcraig, I'll have to read through the discussion link you put up, thanks!

As mentioned, there is no automatic way of doing this but there is no reason that this function could not be added. If you are interested in adding it, we are happy to help, otherwise, we can add it when we have time.

@SaraOgaz Am I right that flags is some kind of unsigned integer array where the bit representation indicates which flags are set?

For example suppose it has 2 flags, i.e. bad pixel, hot pixel. Then a bad pixel is represented by 10, a good pixel by 00 and a bad&hot pixel by 11 (bit-representation).

And the dqbits are some sort of selection criterium like "apply flag 1", "apply no flag" and "apply all flags"? The dqbits would then be a parameter for functions or should they be stored alongside the flags?

@MSeifert04, yeah, that's they way they work for HST data DQ (data quality) arrays. The dqbits would be a parameter, where the total was the bit total for flags you wanted to use for masking, i.e. in ints, I want to flag bit 8 and bit 32, so my dqbits value will be 40.

@SaraOgaz -- could you please provide a sample image that has a DQ array? We want to try to implement this but a concrete example would be helpful. If you are interested in working on a PR that would be great! 😀

It might be a better fit in in the combine function rather than the Combiner class...but in any event a concrete example seems like the place to start.

@mwcraig, if you download any data from HST Archive, you would see such a DQ array. If you have DropBox or something, I can drop one for you.

@pllim I've checked some files from the HST archive but none of those I downloaded had a DQ extension. Could you point me to a file, or instrument or are they only included in pre-processed files, if so in which? I think I get the basic idea and I did some work on that score but I'm still lacking any real-world-data to test it on.

I can't check till Monday. But I'm surprised to hear Archive doesn't give DQ array. What's the dataset name and suffix you requested?

No problem I don't have much time this weekend but I always forgot to ask before about the images. I'll check which files (I requested a lot of random files :) ) then.

@MSeifert04 , here are the instructions to retrieve a specific dataset from HST/ACS that I know for sure has DQ extension. It is a subarray, so will have smaller size than a fullframe exposure.

1. Go to https://archive.stsci.edu/cgi-bin/dataset_lookup
2. Type jc5001soq in top left text area.
3. Click "submit datasets for retrieval".
4. On the next page, click the similar button again. The dataset should already be checked.
5. Now, you can enter your method of retrieval info. By default, you should receive calibrated data. But if you want to be very specific, you can CTRL+click to select FLT and FLC from bottom right menu. FLC is simply FLT that is corrected for CTE loss. Both should have the same format.
6. Click "send retrieval ..." to finalize your request. You should promptly receive an email confirmation.

Hope this helps!

@pllim That worked and has a DQ-extension. Thanks 👍

No problem! In general, any calibrated HST data should have DQ extension. I am curious of the dataset that you got that somehow does not.

@SaraOgaz functions to work with bitmasks have been in stsci.tools for quite some time: https://github.com/spacetelescope/stsci.tools/blob/master/lib/stsci/tools/bitmask.py

>>> from stsci.tools import bitmask
>>> import numpy as np
>>> dqbits = np.asarray([[0,0,1,2,0,8,12,0], [10,4,0,0,0,16,6,0]])
>>> bitmask.bitmask2mask(dqbits, ignore_bits='4,8,16', dtype=int)
array([[1, 1, 0, 0, 1, 1, 1, 1],
       [0, 1, 1, 1, 1, 1, 0, 1]])
>>> bitmask.bitmask2mask(dqbits, ignore_bits='4,8,16', dtype=np.bool)
array([[ True,  True, False, False,  True,  True,  True,  True],
       [False,  True,  True,  True,  True,  True, False,  True]], dtype=bool)
>>>  bitmask.bitmask2mask(dqbits, ignore_bits='4,8,16', good_mask_value=False, dtype=np.bool)
array([[False, False,  True,  True, False, False, False, False],
       [ True, False, False, False, False, False,  True, False]], dtype=bool)

Ability to handle bitmasks have been added to ccdproc have been added, but it still needs to be integrated to be used with combiner

@SaraOgaz -- We're hoping to do get at least some form of this in our 1.3 release (coming up very soon). Would this functionality be useful for now in combining images (doesn't reproduce everything mscombine does):

• If an image fed into combine has a dqarray, that array is flattened to a binary mask.
• the masks are used in doing the combination.

There would not be an output dqmask from this (unless there is a rule we should follow for combining those), but it would at least make it easier to use dqmask with image combination.

Sounds good to me! I'm assuming the flattening let's you pick which bits to respect?

Yeah, the idea would be to use the flatten routine that was added a few months ago, which allows you to specify which bits count.

@SaraOgaz -- punting this to 2.0, but we anticipate releasing that in ~ month, so it shouldn't be too long....

This has taken far longer than anticipated, but on the bright side conversion of bitfields to a boolean mask is in astropy 3.1. I think it would be helpful for us to start to integrate that into CCDData here instead of waiting for integration to happen in astropy. That would also give us a chance to experiment a bit with the interface before fully implementing in astropy.