Improve Q vector generation mechanism

[MBartkowiakSTFC]

Description of work Includes off-diagonal components of HKL indices when finding the extent of HKL values to be used in vector generation.

closes #974

Fixes

1. SphericalLatticeQVectors uses all vector components when checking the HKL range to cover.
2. Q vector generators calculate HKL values of vectors if a unit cell has been provided, even if the vector generator does not use the unit cell definition to generate vectors.
3. IQVectors.create() now takes a UnitCell instead of a Configuration as optional input.
4. Unit tests have been added to check if spherical vectors are generated the same for equivalent unit cell definitions.

To test Unit tests should all pass.

[MBartkowiakSTFC]

The new mechanism of Q vector generation in SphericalLatticeQVectors is this:

1. For each |q|, generate a sphere of vectors with length |q|.
2. Convert all vectors into HKL coordinates (fractional so far) and round them to the nearest integer values.
3. Convert HKL values back to lab frame q vectors, and discard those that are now outside of the width tolerance.
4. Randomly pick the requested number of vectors from the remaining q vectors.

One problem with this approach is that increasing the width does not necessarily increase the spread of |q|, since width is used as tolerance cutoff. The advantage is that we stay as close to spherical distribution of vectors as the unit cell definition allows.

[ChiCheng45]

The new mechanism of Q vector generation in SphericalLatticeQVectors is this:

1. For each |q|, generate a sphere of vectors with length |q|.
2. Convert all vectors into HKL coordinates (fractional so far) and round them to the nearest integer values.
3. Convert HKL values back to lab frame q vectors, and discard those that are now outside of the width tolerance.
4. Randomly pick the requested number of vectors from the remaining q vectors.

One problem with this approach is that increasing the width does not necessarily increase the spread of |q|, since width is used as tolerance cutoff. The advantage is that we stay as close to spherical distribution of vectors as the unit cell definition allows.

Looks good, but are we sure we want to do it this way? This will mean that the width parameter will be different for SphericalLatticeQVectors from all other qvector generators.

[MBartkowiakSTFC]

I'm happy to think about it some more. I could probably check the standard deviation of |q| in a shell, and extent the shell if the spread of the vector length was not large enough. I guess that our users may want to simulate the effects of low q resolution on the results, and then they should get the |q| as smeared as they requested.

[MBartkowiakSTFC]

This PR may be easier to test once #990 has been approved and merged, since we will have the tools to visualise the distribution of vector lengths in individual shells.

[ChiCheng45]

The new mechanism of Q vector generation in SphericalLatticeQVectors is this:

1. For each |q|, generate a sphere of vectors with length |q|.
2. Convert all vectors into HKL coordinates (fractional so far) and round them to the nearest integer values.
3. Convert HKL values back to lab frame q vectors, and discard those that are now outside of the width tolerance.
4. Randomly pick the requested number of vectors from the remaining q vectors.

One problem with this approach is that increasing the width does not necessarily increase the spread of |q|, since width is used as tolerance cutoff. The advantage is that we stay as close to spherical distribution of vectors as the unit cell definition allows.

I was thinking that if we adjust it slightly and do this

1. For each |q|, generate a shell of vectors with length |q| +/- width/2 similarly to SphericalQVectors.
2. Convert all vectors into HKL coordinates (fractional so far) and round them to the nearest integer values.
3. Convert HKL values back to lab frame q vectors.

we might be able to solve the weighting issue seen in https://github.com/ISISNeutronMuon/MDANSE/pull/990#issuecomment-3333945282 and #806. Here is the reasoning: say we have a lattice with the following population,

and if we generate uniformly q-vectors in the shell and then snap them to the reciprocal lattice vectors, we should get a similar number of q-vectors going to each |q| bin from the above. For the red shell the first bin would get more per reciprocal lattice vector. The same argument applies if we have more reciprocal lattice vectors at specific angles, so that it also fixes #806. Maybe it might also be worth updating the q-vector visualizer to plot the distribution in terms of spherical coordinates?

The main problem will be that there will be a bit of spilling of some q-vectors out of the range |q| +/- width/2 but it might be fine.

[ChiCheng45]

One more thing I just realized that we need to thing about.

I think we might want to adjust the q-vector so instead we get a certain amount of q-vectors per shell volume. Note that we currently use a fixed number of vectors for each shells so for the shells with the largest |q| will not be sampled as well as the smaller ones. We adjust need to adjust step 1 from the above so that the q-vector generator to sampled a sphere uniformly.

[MBartkowiakSTFC]

I think we might want to adjust the q-vector so instead we get a certain amount of q-vectors per shell volume. Note that we currently use a fixed number of vectors for each shells so for the shells with the largest |q| will not be sampled as well as the smaller ones. We adjust need to adjust step 1 from the above so that the q-vector generator to sampled a sphere uniformly.

We can think of doing this, but I am not sure if the larger shells really need this level of fine sampling. Probably some convergence testing would help to see if the additional points bring in new information.

If one day we wanted to add error bars to the results, increasing the sampling should reduce the errors even if the results are largely the same.

[ChiCheng45]

I think we might want to adjust the q-vector so instead we get a certain amount of q-vectors per shell volume. Note that we currently use a fixed number of vectors for each shells so for the shells with the largest |q| will not be sampled as well as the smaller ones. We adjust need to adjust step 1 from the above so that the q-vector generator to sampled a sphere uniformly.

We can think of doing this, but I am not sure if the larger shells really need this level of fine sampling. Probably some convergence testing would help to see if the additional points bring in new information.

If one day we wanted to add error bars to the results, increasing the sampling should reduce the errors even if the results are largely the same.

It depends how you set it. We currently could also be sampling the smaller shells we more q-vectors than is needed.

[MBartkowiakSTFC]

Changes since the last time:

1. Vector preview now has two tabs: the statistics of |q| and the vector coordinate distribution of specific shells.
2. Plot creator can create the |q| statistics when a file containing vectors is right-clicked. It can also create the plot of vector coordinates for a specific shell when a shell dataset is right-clicked.
3. Vectors are generated together with weights. For most generators the weight of each vector is 1. For lattice generators, vectors are first randomly generated, and then rounded to the nearest integer value of HKL. If multiple vectors ended up at the same HKL, the number of the initial vectors corresponding to this HKL value is the weights of the vector.
4. DISF, DCSF, EISF and CCF calculations now include vector weights in the calculations. Typically, the rho values for each vector are multiplied by sqrt(weight), and the resulting correlation for each vector ends up being scaled by weight.

[ChiCheng45]

I get an error when I try to "preview vector distribution" in the Actions tab. I see the following error message.

2025-11-25 13:16:35,131 - ERROR - process[36072] - main 36 - EXCEPTION:
<class 'TypeError'>
'configurator' is an unknown keyword argument
<traceback object at 0x000001AA209AC1C0>
Traceback (most recent call last):
  File "C:\Users\...\PycharmProjects\MDANSE\MDANSE_GUI\Src\MDANSE_GUI\InputWidgets\QVectorsWidget.py", line 376, in helper_dialog
    self.helper = self.create_helper()
                  ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\...\PycharmProjects\MDANSE\MDANSE_GUI\Src\MDANSE_GUI\InputWidgets\QVectorsWidget.py", line 367, in create_helper
    dialog_instance = VectorViewer(self._base, configurator=self._configurator)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\...\PycharmProjects\MDANSE\MDANSE_GUI\Src\MDANSE_GUI\InputWidgets\QVectorsWidget.py", line 271, in __init__
    self.shell_panel_3D = ShellPanel(
                          ^^^^^^^^^^^
  File "C:\Users\...\PycharmProjects\MDANSE\MDANSE_GUI\Src\MDANSE_GUI\InputWidgets\QVectorsWidget.py", line 158, in __init__
    super().__init__(*args, **kwargs)
TypeError: 'configurator' is an unknown keyword argument

[MBartkowiakSTFC]

I get an error when I try to "preview vector distribution" in the Actions tab. I see the following error message.

This should be fixed now.

I think that circular vectors don't work correctly at the moment. I'll let you know when that's done.

[MBartkowiakSTFC]

Actually, circular vectors are probably OK, but the 3D scatter plot resets axes scaling in such a way that a circle that should be in a plane appears to be inclined out of plane.

[ChiCheng45]

The spherical plotting looking good. Here is what I see with spericalqvectors and 1000 vectors.

The distribution for both polar and azimuthal angles looks right. To avoid the users getting confused when looking at the polar angle it might be a good idea to normalise these by $1/\sin(\phi)$, $2/ \sin(\phi)$ or something similar. See eq.5 in https://mathworld.wolfram.com/SpherePointPicking.html $0.5 \sin(\phi)$ is the distribution in the polar coordinate.

[ChiCheng45]

Do the vector plots have their weights applied before binning in the histograms? I'm plotting the sphericallattiveqvectors of a cubic cell and I see the following for one of the shells.

There are fewer lattive q-vectors for polar angles around $\pi / 2$, which makes sense. Hopefully, the weights should weight the vectors around $\pi / 2$ more.

[MBartkowiakSTFC]

Do the vector plots have their weights applied before binning in the histograms? I'm plotting the sphericallattiveqvectors of a cubic cell and I see the following for one of the shells.

I don't think that weights were being applied here. Good point, I'll fix it.

[MBartkowiakSTFC]

The distribution for both polar and azimuthal angles looks right. To avoid the users getting confused when looking at the polar angle it might be a good idea to normalise these by 1 / sin ⁡ ( ϕ ) , 2 / sin ⁡ ( ϕ ) or something similar. See eq.5 in https://mathworld.wolfram.com/SpherePointPicking.html 0.5 sin ⁡ ( ϕ ) is the distribution in the polar coordinate.

I am happy to normalise the angles, but won't it cause problems for other, non-spherical vector generators? Then again, maybe the angle distribution is not even useful for most of other vector generators. Do you think this normalisation should be applied always, or just to the spherical ones? I'm not sure what's more confusing in the long run.

[ChiCheng45]

The distribution for both polar and azimuthal angles looks right. To avoid the users getting confused when looking at the polar angle it might be a good idea to normalise these by 1 / sin ⁡ ( ϕ ) , 2 / sin ⁡ ( ϕ ) or something similar. See eq.5 in https://mathworld.wolfram.com/SpherePointPicking.html 0.5 sin ⁡ ( ϕ ) is the distribution in the polar coordinate.

I am happy to normalise the angles, but won't it cause problems for other, non-spherical vector generators? Then again, maybe the angle distribution is not even useful for most of other vector generators. Do you think this normalisation should be applied always, or just to the spherical ones? I'm not sure what's more confusing in the long run.

I think it makes sense to have it always applied. I think if we have a distribution like $\rho(r, \theta, \phi)$, which could be the distribution or density of q-vectors or something, then the count between angles $x$ and $x+C$ would be

N(x + C) - N(x) = \int_{x}^{x+ C} \int_{r_0}^{r_1} \int_{0}^{2\pi} \rho(r, \theta, \phi) r^2 \sin(\theta) d\theta dr d\phi = \int_{x}^{x + C} \tilde{\rho}(\theta)  \sin(\theta)  d\theta

if we make C small then

N(x + C) - N(x) \approx  \frac{d N}{d x} C

and

\int_{x}^{x+ C} \tilde{\rho}(\theta)  \sin(\theta)  d\theta \approx \tilde{\rho}(x)  \sin(x) C

so that the function we are approximating with our histogram is

\frac{d N}{d x} = \tilde{\rho}(x)   \sin(x)

so I think its better to normalise so we plot a histogram which approximates $\tilde{\rho}(x)$.

[MBartkowiakSTFC]

It seems that the main branch generates less shells than this branch, but they are better defined. Here are the statistics for the NaF trajectory (cubic, 21 Angstrom) with q-range reproducing the experiment.

Protos branch:

This branch:

I tried running DISF with the same parameters using SphericalQVectors and SphericalLatticeQVectors. I had to use the third shell, since protos would not generate the first two.

Protos comparison (lattice vectors vs. all vectors):

This branch:

I guess that the next step will be to compare which result is the closest to the experiment.

[ChiCheng45]

I think for the NaF case, protos would be better because it's a liquid system. It's already isotropic, so we won't get any benefit from the better distribution in the angular part. I suppose we need to figure out what the best balance is between a better angular distribution or a more well-defined width in |q|.

[MBartkowiakSTFC]

I tried to make a quick comparison of the experimental data to the calculation, but it probably would have been more meaningful with a better trajectory. Also, I subtracted a constant flat background from the experimental data, and normalised all the results by dividing all points by S(q,0). This is the comparison of DISF results to the experiment, but there are probably too many assumptions that would need to be verified before this comparison can be used for anything.

My current perspective is that:

1. The new sampling of vectors with different weights is a good idea and it should stay in the code.
2. The new relaxed attitude towards the |q| values is too big a change in behaviour to be introduced at this stage, and I would prefer to keep the sharp limits on (q_min, q_max) in each shell. That is, the users will still be able to make broad, smeared and overlapping shells by increasing the width parameter, but setting it to a low value should produce narrow shells.

I will make changes to this PR to make q_width work in a more restrictive way and will then ask you to check it again. Further changes to q-vector generation can be added after the first release.

[MBartkowiakSTFC]

The current procedure of vector generation is:

1. Generate vectors randomly in a specified shape. Typically, we use uniform distribution in angles and normal distribution around the target |q|.
2. Snap/round the vectors to nearest lattice vectors. Weights are determined based on the number of times each vector was repeated in the output.
3. Remove the vectors outside of the |q| range for each shell. This way q_width can be used to control the shell width.
4. Mark the shells that were meant to be created but ended up empty. The calculation results for these shells will be NaN. This is to make sure that the heatmap plots map the |q| values correctly, since it is assumed that the data points are on a uniform grid.

Other changes:

1. Plotter has been adjusted to handle NaN values. For the moment, the areas of heatmaps with NaN values are white.
2. Minor adjustments to plotter layout to reduce the overlap between subplots.
3. Additional corrections to DispersionLatticeQVectors. At the moment these only accept integer start and step values. Also, the supercell input has been removed. (In the end, if we want it at all, then we probably want it for all generators.)
4. Context menu in Plot Creator enables "Vector statistics" plot if you right-click a file that contains vector. To see the vector positions, you have to right-click a specific shell. It will only work for shells that are not empty.

[MBartkowiakSTFC]

I have found one bug which occurs when you load a trajectory, select an instrument in the actions tab and press preview vector distribution. An error is seen in the logs.

2025-12-09 12:21:13,271 - ERROR - process[25588] - Vectors3D 62 - PlottingContext can't plot to None
2025-12-09 12:21:13,271 - ERROR - process[25588] - main 36 - EXCEPTION: <class 'AttributeError'> 'NoneType' object has no attribute 'add_subplot' <traceback object at 0x000002376C950A00>

This should work now.

While fixing this one, I noticed that the new settings of the matplotlib layout engine made "Single" plots look weird if there were too many curves in them. I tried to fix it by setting the figure's layout engine to"none" in the Single plotter.

Also, I added a fix to make the "Vectors" and "Vectors3D" plotters not selectable by the user. This way we can still use them in the code, but the user will not be able to try (and fail) to visualise normal datasets using those.

[ChiCheng45]

Previously width=0 was possible in SphericalQVectors. Its not working anymore, is this intentional?