Usage of 'extract_1d/apply_apcorr.py' stalls program

[kcrasmus]

I am using the jwst module inside of the Eureka pipeline, and I'm getting to a point where the code stalls for hours without an error when it should be proceeding to the next step. I suspect that something, somewhere, is not writing to a header correctly.

Here is the full traceback:

2022-07-29 15:15:23,372 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Applying Aperture correction. 2022-07-29 15:15:29,045 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Extracting integration 12204 2022-07-29 15:15:29,267 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Applying Aperture correction. 2022-07-29 15:15:30,418 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - All 12204 integrations done ^CTraceback (most recent call last): File "/Users/rasmussen/Research/WORKING_Eureka/demos/run_JN_eureka.py", line 33, in #s2_meta = s2.calibrateJWST(eventlabel, ecf_path=ecf_path) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/eureka/S2_calibrations/s2_calibrate.py", line 159, in calibrateJWST pipeline.run_eurekaS2(filename, meta, log) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/eureka/S2_calibrations/s2_calibrate.py", line 252, in run_eurekaS2 self(filename) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stpipe/step.py", line 485, in run step_result = self.process(*args) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 117, in process result = self.process_exposure_product( File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 307, in process_exposure_product x1d = self.extract_1d(resampled) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stpipe/step.py", line 529, in run self.save_model(result, idx=idx, format=self.name_format) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stpipe/step.py", line 993, in save_model output_path = model.save( File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/model_base.py", line 523, in save self.to_fits(output_path, *args, **kwargs) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/model_base.py", line 635, in to_fits hdulist, tree = fits_support.to_fits(self._instance, self._schema) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 433, in to_fits _save_from_schema(hdulist, tree, schema) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 369, in _save_from_schema validator.validate(tree, _schema=schema) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jsonschema/validators.py", line 267, in validate for error in self.iter_errors(*args, **kwargs): File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/asdf/schema.py", line 315, in iter_errors yield from super(ASDFValidator, self).iter_errors(instance) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jsonschema/validators.py", line 242, in iter_errors for error in errors: File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 219, in _fits_comment_section_handler for error in validator.descend( File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jsonschema/validators.py", line 259, in descend for error in self.evolve(schema=schema).iter_errors(instance): File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/asdf/schema.py", line 315, in iter_errors yield from super(ASDFValidator, self).iter_errors(instance) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jsonschema/validators.py", line 242, in iter_errors for error in errors: File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 296, in _fits_item_recurse for error in validator.descend(item, items, path=index): File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jsonschema/validators.py", line 259, in descend for error in self.evolve(schema=schema).iter_errors(instance): File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/asdf/schema.py", line 315, in iter_errors yield from super(ASDFValidator, self).iter_errors(instance) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jsonschema/validators.py", line 242, in iter_errors for error in errors: File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 219, in _fits_comment_section_handler for error in validator.descend( File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jsonschema/validators.py", line 259, in descend for error in self.evolve(schema=schema).iter_errors(instance): File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/asdf/schema.py", line 315, in iter_errors yield from super(ASDFValidator, self).iter_errors(instance) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/jsonschema/validators.py", line 241, in iter_errors errors = validator(self, v, instance, _schema) or () File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 237, in _fits_element_writer hdu = _get_or_make_hdu(fits_context.hdulist, hdu_name, index=fits_context.sequence_index) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 180, in _get_or_make_hdu hdu = get_hdu(hdulist, hdu_name, index=index) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 134, in get_hdu hdu = hdulist[pair] File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/io/fits/hdu/hdulist.py", line 322, in getitem self._positive_index_of(key)) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/io/fits/hdu/hdulist.py", line 777, in _positive_index_of index = self.index_of(key) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/io/fits/hdu/hdulist.py", line 753, in index_of (_ver is None or _ver == hdu.ver)): File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/io/fits/hdu/base.py", line 209, in ver return self._header.get('EXTVER', 1) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/io/fits/header.py", line 887, in get return self[key] File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/io/fits/header.py", line 163, in getitem value = card.value File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/io/fits/card.py", line 293, in value if conf.strip_header_whitespace and isinstance(value, str): File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/config/configuration.py", line 303, in get return self() File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/config/configuration.py", line 430, in call options.append((sec[self.name], self.module, self.name)) File "/Users/rasmussen/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/extern/configobj/configobj.py", line 550, in getitem def getitem(self, key): KeyboardInterrupt

[tapastro]

Is the error reproducible when running your data through the standalone jwst pipeline?

If not, you may want to file an issue against Eureka.

[kcrasmus]

Hello,

Yes, when I run 'strun calwebb_spec2

Stage1/jw02420001001_04101_00001-seg001_nrs1_rateints.fits ', I am getting the same stalling behavior after it completes ...

2022-08-01 16:06:34,549 - stpipe.Spec2Pipeline.extract_1d - INFO - Extracting integration 12204

2022-08-01 16:06:34,609 - stpipe.Spec2Pipeline.extract_1d - INFO - Applying Aperture correction.

2022-08-01 16:06:35,010 - stpipe.Spec2Pipeline.extract_1d - INFO - All 12204 integrations done

Thanks, Kaitlin

On Mon, Aug 1, 2022 at 11:11 AM Tyler Pauly @.***> wrote:

Is the error reproducible when running your data through the standalone jwst pipeline?

— Reply to this email directly, view it on GitHub https://github.com/spacetelescope/jwst/issues/6956#issuecomment-1201542541, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2E4CUFB77J6UDLD54LPZYLVXAHMTANCNFSM55IM7QLA . You are receiving this because you authored the thread.Message ID: @.***>

[tapastro]

Hi Kaitlin,

Sorry that I never followed up, but the stalling you encountered was likely due to the extract_1d step saving your 12204 integrations into the x1dints.fits file, which must be fairly large. Do you still encounter this issue? If so, do you see the same stalling behavior with inputs of <100 integrations?