lightkurve
lightkurve copied to clipboard
Published
20 hours ago •
lightkurve
lightkurve
More `MaskedNDArray` failures with Astropy 5.0
Problem description
Found a couple more test failures with astropy 5.0rc2 that I missed previously (likely were present with 5.0rc1 as well, but I did not test with remote-data). All passing with 4.3.1 in the same environment; I have not looked into details, but they all seem in some way related to handling MaskedNDArray (rather than unmasked NDData or np.ma.array) now.
The first test e.g. could be fixed by substituting
assert np.isnan(getattr(lc.flux, 'data', lc.flux)).any() # ensure the test data has nan in flux
Example
poetry run pytest --remote-data
______________________________________________________________________ test_preprocess_lc ______________________________________________________________________
@pytest.mark.remote_data
def test_preprocess_lc():
"""Test to ensure the lightcurve is pre-processed before applying BLS for correctness and consistent output"""
from lightkurve.interact_bls import _preprocess_lc_for_bls
lc = KeplerLightCurve.read(KEPLER10)
> assert np.isnan(lc.flux).any() # ensure the test data has nan in flux
E assert MaskedNDArray(False)
E + where MaskedNDArray(False) = <bound method MaskedNDArray.any of MaskedNDArray([ ———, False, False, ..., False, False, False])>()
E + where <bound method MaskedNDArray.any of MaskedNDArray([ ———, False, False, ..., False, False, False])> = MaskedNDArray([ ———, False, False, ..., False, False, False]).any
E + where MaskedNDArray([ ———, False, False, ..., False, False, False]) = <ufunc 'isnan'>(<MaskedQuantity [ ———, 575100.6 , 575187.25, ..., 575059.7 ,\n 575059.3 , 574996.6 ] electron / s>)
E + where <ufunc 'isnan'> = np.isnan
E + and <MaskedQuantity [ ———, 575100.6 , 575187.25, ..., 575059.7 ,\n 575059.3 , 574996.6 ] electron / s> = <KeplerLightCurve length=1011 LABEL="KIC 11904151" QUARTER=4 AUTHOR=Kepler FLUX_ORIGIN=pdcsap_flux>\n time ...58 5.7505931e+05 ... 6.4287381e-03 4.3517463e-02\n373.21904601071583 5.7499662e+05 ... 6.4082127e-03 4.3601818e-02.flux
tests/test_interact_bls.py:85: AssertionError
____________________________________________________________________ test_lightcurve_plots _____________________________________________________________________
@pytest.mark.remote_data
def test_lightcurve_plots():
"""Sanity check to verify that lightcurve plotting works"""
for lc in [KeplerLightCurve.read(TABBY_Q8), TessLightCurve.read(TESS_SIM)]:
lc.plot()
lc.scatter()
> lc.errorbar()
tests/test_lightcurve.py:416:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:1999: in errorbar
return self._create_plot(method="errorbar", **kwargs)
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:1849: in _create_plot
ax.errorbar(
/opt/lib/python3.9/site-packages/matplotlib/__init__.py:1412: in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
/opt/lib/python3.9/site-packages/matplotlib/axes/_axes.py:3415: in errorbar
barcols.append(lines_func(
/opt/lib/python3.9/site-packages/matplotlib/__init__.py:1412: in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
/opt/lib/python3.9/site-packages/matplotlib/axes/_axes.py:1125: in vlines
x, ymin, ymax = cbook._combine_masks(x, ymin, ymax)
/opt/lib/python3.9/site-packages/matplotlib/cbook/__init__.py:1032: in _combine_masks
x = safe_masked_invalid(x)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
x = MaskedNDArray([ ———, 258310.04830265, 258301.8428278 , ...,
258854.41510582, 258792.36798096, 258795.49073887]), copy = False
def safe_masked_invalid(x, copy=False):
x = np.array(x, subok=True, copy=copy)
if not x.dtype.isnative:
# If we have already made a copy, do the byteswap in place, else make a
# copy with the byte order swapped.
x = x.byteswap(inplace=copy).newbyteorder('N') # Swap to native order.
try:
xm = np.ma.masked_invalid(x, copy=False)
> xm.shrink_mask()
E AttributeError: 'MaskedNDArray' object has no attribute 'shrink_mask'
/opt/lib/python3.9/site-packages/matplotlib/cbook/__init__.py:708: AttributeError
___________________________________________________________________ test_lightcurve_scatter ____________________________________________________________________
@pytest.mark.remote_data
def test_lightcurve_scatter():
"""Sanity check to verify that lightcurve scatter plotting works"""
lc = KeplerLightCurve.read(KEPLER10)
> lc = lc.flatten()
tests/test_lightcurve.py:434:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:813: in flatten
mask &= np.isfinite(self.flux)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = MaskedNDArray([ ———, True, True, ..., True, True, True]), ufunc = <ufunc 'bitwise_and'>, method = '__call__'
inputs = (array([ True, True, True, ..., True, True, True]), MaskedNDArray([ ———, True, True, ..., True, True, True])), kwargs = {}
out = (array([ True, True, True, ..., True, True, True]),), out_unmasked = (array([ True, True, True, ..., True, True, True]),), out_mask = None
out_masks = (None,), d = array([ True, True, True, ..., True, True, True]), m = None
def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
out = kwargs.pop('out', None)
out_unmasked = None
out_mask = None
if out is not None:
out_unmasked, out_masks = self._get_data_and_masks(*out)
for d, m in zip(out_unmasked, out_masks):
if m is None:
# TODO: allow writing to unmasked output if nothing is masked?
if d is not None:
> raise TypeError('cannot write to unmasked output')
E TypeError: cannot write to unmasked output
/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:659: TypeError
_______________________________________________________________________ test_cdpp_tabby ________________________________________________________________________
@pytest.mark.remote_data
def test_cdpp_tabby():
"""Compare the cdpp noise metric against the pipeline value."""
lc = KeplerLightCurve.read(TABBY_Q8)
# Tabby's star shows dips after cadence 1000 which increase the cdpp
lc2 = LightCurve(time=lc.time[:1000], flux=lc.flux[:1000])
> assert np.abs(lc2.estimate_cdpp().value - lc.cdpp6_0) < 30
tests/test_lightcurve.py:486:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:1573: in estimate_cdpp
detrended_lc = self.flatten(
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:813: in flatten
mask &= np.isfinite(self.flux)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = MaskedNDArray([ ———, True, True, True, True, True, True, True,
True, True, True, True, Tru...True, True, True, True, True, True, True,
True, True, True, True, True, True, True, True])
ufunc = <ufunc 'bitwise_and'>, method = '__call__'
inputs = (array([ True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True...rue, True, True, True, True, True, True,
True, True, True, True, True, True, True, True]))
kwargs = {}
out = (array([ True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True...rue, True, True, True, True,
True, True, True, True, True, True, True, True, True,
True]),)
out_unmasked = (array([ True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True...rue, True, True, True, True,
True, True, True, True, True, True, True, True, True,
True]),)
out_mask = None, out_masks = (None,)
d = array([ True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True,... True, True, True, True, True,
True, True, True, True, True, True, True, True, True,
True])
m = None
def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
out = kwargs.pop('out', None)
out_unmasked = None
out_mask = None
if out is not None:
out_unmasked, out_masks = self._get_data_and_masks(*out)
for d, m in zip(out_unmasked, out_masks):
if m is None:
# TODO: allow writing to unmasked output if nothing is masked?
if d is not None:
> raise TypeError('cannot write to unmasked output')
E TypeError: cannot write to unmasked output
/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:659: TypeError
_____________________________________________________________________ test_regression_346 ______________________________________________________________________
@pytest.mark.remote_data # due to test (K2_C08 is remote)
def test_regression_346():
"""Regression test for https://github.com/lightkurve/lightkurve/issues/346"""
# This previously triggered an IndexError:
with warnings.catch_warnings(): # KeplerLightCurveFile is deprecated
warnings.simplefilter("ignore", LightkurveDeprecationWarning)
from lightkurve import KeplerLightCurveFile
> KeplerLightCurveFile(
K2_C08
).PDCSAP_FLUX.remove_nans().to_corrector().correct().estimate_cdpp()
tests/test_lightcurve.py:1107:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../opt/lib/python3.9/site-packages/lightkurve/correctors/sffcorrector.py:173: in correct
ar[~np.in1d(ar, ar[a:b])] = 0
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = (MaskedNDArray([0.88009785, 0.99366021, 1.11913694, ..., 0.63925245,
0.72696675, 0.79226021]), MaskedND...68, 1.09270334,
1.17217677, 1.25490099, 1.3383802 , 1.41997026, 1.50608308,
1.59696806]))
kwargs = {}
relevant_args = (MaskedNDArray([0.88009785, 0.99366021, 1.11913694, ..., 0.63925245,
0.72696675, 0.79226021]), MaskedND...68, 1.09270334,
1.17217677, 1.25490099, 1.3383802 , 1.41997026, 1.50608308,
1.59696806]))
> ???
E TypeError: no implementation found for 'numpy.in1d' on types that implement __array_function__: [<class 'astropy.utils.masked.core.MaskedNDArray'>]
<__array_function__ internals>:5: TypeError
_________________________________________________________________ test_CBVCorrector_retrieval __________________________________________________________________
@pytest.mark.remote_data
def test_CBVCorrector_retrieval():
"""Tests CBVCorrector by retrieving some sample Kepler/TESS light curves
and correcting them
"""
# ***
# A good TESS example of both over- and under-fitting
# The "over-fitted" curve looks better to the eye, but eyes can be deceiving!
lc = search_lightcurve(
"TIC 357126143", mission="tess", author="spoc", sector=10
).download(flux_column="sap_flux")
cbvCorrector = CBVCorrector(lc)
assert isinstance(cbvCorrector, CBVCorrector)
cbv_type = ["SingleScale", "Spike"]
cbv_indices = [np.arange(1, 9), "ALL"]
# Gaussian Prior correction
> lc = cbvCorrector.correct_gaussian_prior(
cbv_type=cbv_type, cbv_indices=cbv_indices, alpha=1e-2
)
tests/correctors/test_cbvcorrector.py:452:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../opt/lib/python3.9/site-packages/lightkurve/correctors/cbvcorrector.py:278: in correct_gaussian_prior
self.correct_regressioncorrector(self.design_matrix_collection,
../../opt/lib/python3.9/site-packages/lightkurve/correctors/cbvcorrector.py:501: in correct_regressioncorrector
return super(CBVCorrector, self).correct(design_matrix_collection, **kwargs)
../../opt/lib/python3.9/site-packages/lightkurve/correctors/regressioncorrector.py:257: in correct
self.outlier_mask |= sigma_clip(residuals, sigma=sigma).mask
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:835: in sigma_clip
return sigclip(data, axis=axis, masked=masked,
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:638: in __call__
return self._sigmaclip_noaxis(data, masked=masked,
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:420: in _sigmaclip_noaxis
self._compute_bounds(filtered_data, axis=None)
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:302: in _compute_bounds
self._max_value = self._cenfunc_parsed(data, axis=axis)
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:69: in _nanmedian
return array.__array_wrap__(bottleneck.nanmedian(array, axis=axis))
/opt/lib/python3.9/site-packages/astropy/units/quantity.py:563: in __array_wrap__
return self._new_view(obj)
/opt/lib/python3.9/site-packages/astropy/units/quantity.py:746: in _new_view
view.__array_finalize__(self)
/opt/lib/python3.9/site-packages/astropy/units/quantity.py:540: in __array_finalize__
super_array_finalize(obj)
/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:557: in __array_finalize__
self._set_mask(getattr(obj, '_mask', False))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <MaskedQuantity 72.30110983 electron / s>, mask = array([False, False, False, ..., False, False, False]), copy = False
def _set_mask(self, mask, copy=False):
self_dtype = getattr(self, 'dtype', None)
mask_dtype = (np.ma.make_mask_descr(self_dtype)
if self_dtype and self_dtype.names else np.dtype('?'))
ma = np.asanyarray(mask, dtype=mask_dtype)
if ma.shape != self.shape:
# This will fail (correctly) if not broadcastable.
self._mask = np.empty(self.shape, dtype=mask_dtype)
> self._mask[...] = ma
E ValueError: could not broadcast input array from shape (17692,) into shape ()
/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:226: ValueError
--------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------
Started querying MAST for observations with the exact target_name='357126143'.
MAST found 1 observations. Now querying MAST for the corresponding data products.
MAST found 1 matching data products.
File found in local cache.
---------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
DEBUG lightkurve.search:search.py:1121 Started querying MAST for observations with the exact target_name='357126143'.
DEBUG lightkurve.search:search.py:939 MAST found 1 observations. Now querying MAST for the corresponding data products.
DEBUG lightkurve.search:search.py:1001 MAST found 1 matching data products.
DEBUG lightkurve.search:search.py:281 File found in local cache.
______________________ test_remote_data[https://archive.stsci.edu/missions/k2/lightcurves/c8/220100000/39000/ktwo220139473-c08_llc.fits] _______________________
path = 'https://archive.stsci.edu/missions/k2/lightcurves/c8/220100000/39000/ktwo220139473-c08_llc.fits'
@pytest.mark.remote_data
@pytest.mark.parametrize("path", [K2_C08])
def test_remote_data(path):
"""Can we correct a simple K2 light curve?"""
lc = KeplerLightCurve.read(path, quality_bitmask=None)
sff = SFFCorrector(lc.remove_nans())
> sff.correct(windows=10, bins=5, timescale=0.5)
tests/correctors/test_sffcorrector.py:31:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../opt/lib/python3.9/site-packages/lightkurve/correctors/sffcorrector.py:173: in correct
ar[~np.in1d(ar, ar[a:b])] = 0
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = (MaskedNDArray([0.88009785, 0.99366021, 1.11913694, ..., 0.63925245,
0.72696675, 0.79226021]), MaskedND...35,
1.13949553, 1.2075622 , 1.27598666, 1.3474425 , 1.41329164,
1.47335664, 1.53116095]))
kwargs = {}
relevant_args = (MaskedNDArray([0.88009785, 0.99366021, 1.11913694, ..., 0.63925245,
0.72696675, 0.79226021]), MaskedND...35,
1.13949553, 1.2075622 , 1.27598666, 1.3474425 , 1.41329164,
1.47335664, 1.53116095]))
> ???
E TypeError: no implementation found for 'numpy.in1d' on types that implement __array_function__: [<class 'astropy.utils.masked.core.MaskedNDArray'>]
<__array_function__ internals>:5: TypeError
____________________________________________________________________ test_sff_nan_centroids ____________________________________________________________________
@pytest.mark.remote_data
def test_sff_nan_centroids():
"""Regression test for #827: SFF failed if light curve contained
NaNs in its `centroid_col` or `centroid_row` columns."""
lc = search_lightcurve("EPIC 211083408", author="K2").download()
# This previously raised a ValueError:
> lc[200:500].remove_nans().to_corrector("sff").correct()
tests/correctors/test_sffcorrector.py:237:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../opt/lib/python3.9/site-packages/lightkurve/correctors/sffcorrector.py:173: in correct
ar[~np.in1d(ar, ar[a:b])] = 0
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = (MaskedNDArray([1.74682912, 1.97256351, 2.73136771, 2.73306766, 2.7346456 ,
2.73633047, 2.73865168, 2.7...33047, 2.73865168, 2.74123547, 2.74447127, 2.74615955,
2.73309043, 2.71391645, 2.72493336, 2.72557768]))
kwargs = {}
relevant_args = (MaskedNDArray([1.74682912, 1.97256351, 2.73136771, 2.73306766, 2.7346456 ,
2.73633047, 2.73865168, 2.7...33047, 2.73865168, 2.74123547, 2.74447127, 2.74615955,
2.73309043, 2.71391645, 2.72493336, 2.72557768]))
> ???
E TypeError: no implementation found for 'numpy.in1d' on types that implement __array_function__: [<class 'astropy.utils.masked.core.MaskedNDArray'>]
<__array_function__ internals>:5: TypeError
--------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------
Started querying MAST for observations with the exact target_name='ktwo211083408'.
MAST found 1 observations. Now querying MAST for the corresponding data products.
MAST found 1 matching data products.
File found in local cache.
---------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
DEBUG lightkurve.search:search.py:1121 Started querying MAST for observations with the exact target_name='ktwo211083408'.
DEBUG lightkurve.search:search.py:939 MAST found 1 observations. Now querying MAST for the corresponding data products.
DEBUG lightkurve.search:search.py:1001 MAST found 1 matching data products.
DEBUG lightkurve.search:search.py:281 File found in local cache.
____________________________________________________________________ test_asteroseismology _____________________________________________________________________
@pytest.mark.remote_data
def test_asteroseismology():
datalist = search_lightcurve("KIC11615890")
data = datalist.download_all()
> lc = data[0].normalize().flatten()
tests/seismology/test_butler.py:16:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:813: in flatten
mask &= np.isfinite(self.flux)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = MaskedNDArray([ True, True, True, True, True, True, True, True,
True, True, True, True, Tru...rue, True, True, True,
True, True, True, True, True, True, True, True,
True])
ufunc = <ufunc 'bitwise_and'>, method = '__call__'
inputs = (array([ True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True...ue, True, True, True,
True, True, True, True, True, True, True, True,
True]))
kwargs = {}
out = (array([ True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True...rue,
True, True, True, True, True, True, True, True, True,
True, True, True, True, True]),)
out_unmasked = (array([ True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True...rue,
True, True, True, True, True, True, True, True, True,
True, True, True, True, True]),)
out_mask = None, out_masks = (None,)
d = array([ True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True,... True,
True, True, True, True, True, True, True, True, True,
True, True, True, True, True])
m = None
def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
out = kwargs.pop('out', None)
out_unmasked = None
out_mask = None
if out is not None:
out_unmasked, out_masks = self._get_data_and_masks(*out)
for d, m in zip(out_unmasked, out_masks):
if m is None:
# TODO: allow writing to unmasked output if nothing is masked?
if d is not None:
> raise TypeError('cannot write to unmasked output')
E TypeError: cannot write to unmasked output
/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:659: TypeError
[skipping warnings]
=================================================================== short test summary info ====================================================================
FAILED tests/test_interact_bls.py::test_preprocess_lc - assert MaskedNDArray(False)
FAILED tests/test_lightcurve.py::test_lightcurve_plots - AttributeError: 'MaskedNDArray' object has no attribute 'shrink_mask'
FAILED tests/test_lightcurve.py::test_lightcurve_scatter - TypeError: cannot write to unmasked output
FAILED tests/test_lightcurve.py::test_cdpp_tabby - TypeError: cannot write to unmasked output
FAILED tests/test_lightcurve.py::test_regression_346 - TypeError: no implementation found for 'numpy.in1d' on types that implement __array_function__: [<clas...
FAILED tests/correctors/test_cbvcorrector.py::test_CBVCorrector_retrieval - ValueError: could not broadcast input array from shape (17692,) into shape ()
FAILED tests/correctors/test_sffcorrector.py::test_remote_data[https://archive.stsci.edu/missions/k2/lightcurves/c8/220100000/39000/ktwo220139473-c08_llc.fits]
FAILED tests/correctors/test_sffcorrector.py::test_sff_nan_centroids - TypeError: no implementation found for 'numpy.in1d' on types that implement __array_fu...
FAILED tests/seismology/test_butler.py::test_asteroseismology - TypeError: cannot write to unmasked output
======================================== 9 failed, 315 passed, 26 skipped, 4 xfailed, 325 warnings in 435.04s (0:07:15) ========================================
Environment
Platform: macOS-10.14.6-x86_64-i386-64bit Python 3.9.8, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 Astropy: 5.0rc2 Matplotlib: 3.5.0rc1 Freetype: 2.6.1 rootdir: ~/git/lightkurve, configfile: pyproject.toml, testpaths: tests, src plugins: filter-subpackage-0.1.1, arraydiff-0.3, xdist-2.1.0, astropy-header-0.1.2, cov-2.10.1, flaky-3.7.0, remotedata-0.3.2, mock-3.3.1, asdf-2.8.1, forked-1.3.0, openfiles-0.5.0, mpl-0.11, expect-1.1.0, hypothesis-6.14.2, doctestplus-0.10.1, flake8-1.0.6
lightkurve version: 2.0.12dev Installation method: git main @ 70d1c4c
Thanks for reporting this @dhomeier! It looks like 9 separate tests failed in total. We'll probably have to debug one by one.
I'm just going to loop @mhvk into this issue as I wonder if some of these are issues with the new masked quantity class?
Several of the failing tests can be narrowed down to the TypeError demonstrated by the example below. It's not immediately obvious to me whether this is a bug or a feature to keep me safe. Thoughts?
>>> import numpy as np
>>> from astropy.utils.masked import Masked
>>> notmasked = np.zeros(3, dtype=bool)
>>> masked = Masked(notmasked)
>>> notmasked &= masked
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/var/folders/x4/y38zs5fj4szf7xj3354kv74m0000gn/T/ipykernel_23575/1983679778.py in <module>
3 notmasked = np.zeros(3, dtype=bool)
4 masked = Masked(notmasked)
----> 5 notmasked &= masked
~/Library/Caches/pypoetry/virtualenvs/lightkurve-HqDeGDfc-py3.8/lib/python3.8/site-packages/astropy/utils/masked/core.py in __array_ufunc__(self, ufunc, method, *inputs, **kwargs)
657 # TODO: allow writing to unmasked output if nothing is masked?
658 if d is not None:
--> 659 raise TypeError('cannot write to unmasked output')
660 elif out_mask is None:
661 out_mask = m
TypeError: cannot write to unmasked output
I think it's a decision to err on the safe side since the output could not be written to unmasked without discarding the mask. As the comment states, there is no exception yet even for masked.mask.any() == False.
I think for this to work in its current state, mask in LightCurve.flatten would need to be initialised as Masked(np.ones(len(self.time), dtype=bool)) if astropy is at 5.x/has utils.masked.Masked, perhaps conditional on self.flux being of masked type. This would go beyond just adapting the tests of course, but should resolve 3 of the failures, and will probably be necessary to work safely with the new MaskedNDData.
@barentsen - indeed, I was looking and thinking similarly. As @dhomeier notes, Masked prefers not to discard the mask, unlike numpy's MaskedArray, which does discard the mask in those operations. For that piece of code, I think there are two options:
- Let numpy eat the mask by doing
mask[~np.isfinite(self.flux)] = False(this might actually be faster...) - Use
flux = self.flux.filled(np.nan)and then work withfluxfrom thereon.
p.s. Just to be sure: is the whole np.isfinite call needed? Shouldn't all non-finite values already be masked? If so, one can just use the mask, i.e., something like mask &= self.flux.mask if hasattr(self.flux, 'mask') else np.isfinite(self.flux)
EDIT: I think the mask as a pure ndarray, as you have it in flatten, is most logical, so would try to keep that!
On np.in1d - oops - see https://github.com/astropy/astropy/blob/b740594dcc9f8aacae0c14a5774e09f0732d1aef/astropy/utils/masked/function_helpers.py#L140-L142
p.s. Just to be sure: is the whole np.isfinite call needed? Shouldn't all non-finite values already be masked?
Excellent question! Of course most of this code was written in a world where MaskedQuantity didn't exist yet, hence lots of np.isfinite() everywhere. I expect we'll be able to resolve these failing tests with trivial fixes for now however, stay tuned!
Great! Feel free to ping if there are issues. I'll look into np.in1d, since it would be nice to support that and the other set-like functions -- but I may have to think a bit exactly what a masked entry means for some of the arguments.
@mhvk I noticed the following issue while debugging one of the tests: https://github.com/astropy/astropy/issues/12481
(It's easy enough to work around this, but it looked like a genuine albeit obscure issue to me.)
@mhvk @barentsen - just to check, should I consider the issues here blockers for the astropy 5.0 release or can I go ahead and release and we fix these in 5.0.1?
@astrofrog I haven't seen anything that should block a 5.0 release from my perspective!