UVData.write_ms(...reflect_conj: bool) reflect MS

[radonnachie]

I've produced UVH5 files, the MS of which results in reflected images in CASA. The fix was to reproduce the UVH5 file with a negation in the uvw_array (flipping the baseline direction). This breaks the file's self-consistency regarding baseline convention, and pyuvdata complains about the mismatch: The uvw_array does not match the expected values given the antenna positions. The largest discrepancy is 16996.07144383784 meters.

I'd like to not produce inconsistent UVH5 files and thus uphold the convention standard of the file. The issue is more squarely on CASA's side of my operation after all, not the UVH5 file.

The proposal is to provide a different flag when writing the MS, one that does not negate the uvw_array AND conjugate the visdata, but either or, producing a reflected MS.

I broach this first as an issue to verify it's validity. Will happily open the appropriate PR at positive response.

[kartographer]

@radonnachie -- thank you for the report, can I ask if you've looked at all as to whether or not the visibility data are conjugated in the expected/correct way? Nominally the uvw-convention (ant1 - ant2 versus ant2 - ant1) is linked to the conjugation scheme of the data, and the above would suggest a mismatch between that convention and the uvws being calculated.

[radonnachie]

Ah, I did omit that piece of info: I double checked my code to ensure consistency (uvw is ant2-ant1 and ant2 is conj, and further that the index order is consistent between ant_?_array and the correlation) in the convention and don't see an issue.

Is there a visual inspection of the values that I would be able to perform 🤔 the auto-correlations are all real but that isn't very telling... I'm leaning on warnings from pyuvdata to infer correctness and convention consistency, as I implied above.

[kartographer]

There's not a super easy check to do aside from the imaging check you've already done, which does confirm that somewhere a conjugation flip is happening, and that either the uvw_array values need to be negated or data_array needs to be complex-conjugated. I did look through the code, and there's nothing obvious that looks off, though without knowing more about some instrument-specific details, it's difficult to tell if everything is "right" (since there's lots of places where a conjugation can sneak in along a signal chain). But from what you've reported thus far, it may actually be that you've got an internal consistency issue with the data itself, rather than a convention with a particular file format. If one wanted to be aligned to the convention that pyuvdata uses of ant2-ant1 versus ant1-ant2), then the visibilities would need need to be complex conjugated prior to be written to disk.

On a related note, we do actually have some code that looks for the convention flip and attempts to automatically fix it -- it was introduced because of the mixed convention that CASA uses (documented one way, but supports ALMA another way...), which looks at the values in uvw_array and if it sees that the negated version agrees well with what is expected, will automatically do this flip under the hood. That you're seeing the error would suggest that the uvw-values are not quite matching what is expected which may be worth investigating further. Happy to help with this if you're looking for a hand, and can communicate more about this offline.

[nmahesh1412]

Is there an update on including this flag while writing out an MS file?

[kartographer]

So following up here -- I believe what sparked @nmahesh1412's comment here turned out to be what I'm going to call a bug w/ pyuvsim (see https://github.com/RadioAstronomySoftwareGroup/pyuvsim/issues/473), where the conjugation flip would not have fully fixed the problem at hand (though admittedly would have been pretty close, but far from typical UVData tolerances).

As I mentioned above, I'm pretty sure there's no convention by which the visibility conjugation would be flipped alone (i.e., without negating from the uvws), so the above is basically trying to fix the data but only after writing to a MS file, which I don't think is the right approach here. The more straightforward thing to do is to manipulate the data array directly when it's needed (i.e., uvd.data_array = np.conj(uvd.data_array). I could see maybe writing a utility function for doing this if for some reason there was concern about directly accessing the underlying parameters (or maybe what's actually needed is a helper script to fix this in the underlying file being read in, but that's a bigger task).

I'm going to close this for now since I think this isn't actually a pyuvdata issue, nor something that calls for additional functionality, but we can re-open it as needed if others have more thoughts.