Drizzlepac/HAP: Refactoring poller_utils.py

[stscijgbot-hstdp]

trafficstars

Issue HLA-1153 was created on JIRA by Steve Goldman:

Proposal for updating poller_utils.py

 

Currently the main benefit to using full poller files is that we can specify the location of the file. This, however, can be achieved with another new streamlined system. It was thought that there was also the added benefit that a full poller file does not read in the data and so it is faster. However, the sort_poller_table (poller_utils line:1472) reads the header of every file anyway, and so this is not actually one of the benefits. I proposed that we use the poller file in a different manner.

 

I propose that poller files have new commented out column names, and that the only critical column is the filename for SVMs and the filename and skycell_name for MVMs; a skycell_name column would signify an MVM product.  Any additional column added to the poller file will be added to the Product Objects. Any column name ending in "_update" would be used to update header keywords with the same name as the poller file column name (minus the "_update").

 

Within the code, the poller table will be read-in as an astropy table, the table will be passed through the code, adding flags, new parsed keywords, etc. The table will then be passed to a code that will verify the table, and then this table will be passed to the various Product Objects.

 

This creates several benefits:

• the poller files can input data in a uniform way (current method is effectively 6 different ways; [SVM, MVM] x [simple poller, update-header poller, full poller])
• the table can be read-in without the need to parse strings of varying lengths and formats
• the astropy table can be more-easily inspected and tested
• the code can be re-written with test-driven development in mind
• most importantly the code is currently so opaque that any improvement take major changes to the structure of input interface for all 6 methods.

Example of the current format for a 'full' poller file:

j8ph03iuq_flc.fits,9774,8PH,03,90.0,CLEAR1L;F814W,WFC,/ifs/archive/dev/hst/store/dmsauto/owl_auto/test_dmsauto_20231010_02/regr_08129/pipe/repro_sci/hap/acs_8ph_03/j8ph03iuq_flc.fits```