CI Optimization and Symmetry Loss

[NastaMauger]

Is your feature request related to a problem? Please describe. I'm working on benzene and optimizing CI coefficients from a CAS(6,6) calculation. Before optimization, I had several degenerate CSF coefficients due to D6h symmetry. However, after optimization, these coefficients become unequal, indicating that the procedure breaks symmetry.

Describe the solution you'd like An option to constrain the CI coefficient optimization to preserve degeneracies, especially in symmetric systems. This would help maintain wavefunction symmetry and lead to more consistent and physically meaningful results.

Describe alternatives you've considered One attempted workaround involved combining degenerate CSFs into a single CSF with multiple determinants. However, this manual modification led to noticeably different total energies and degraded the performance of cusp corrections, indicating that it is not a reliable solution.

Unmodified xml before optimization:

          <csf id="CSFcoeff_1" exctLvl="2" coeff="-0.14912" qchem_coeff="-0.14912" occ="220020">
            <det id="csf_1-0" coeff="1" alpha="110010" beta="110010"/>
          </csf>
          <csf id="CSFcoeff_2" exctLvl="2" coeff="-0.14912" qchem_coeff="-0.14912" occ="202200">
            <det id="csf_2-0" coeff="1" alpha="101100" beta="101100"/>
          </csf>

Unmodified xml after optimization:

          <csf id="CSFcoeff_1" exctLvl="2" coeff="-9.2093805577755899e-02" qchem_coeff="-0.14912" occ="220020">
            <det id="csf_1-0" coeff="1" alpha="110010" beta="110010"/>
          </csf>
          <csf id="CSFcoeff_2" exctLvl="2" coeff="-8.9811449518137215e-02" qchem_coeff="-0.14912" occ="202200">
            <det id="csf_2-0" coeff="1" alpha="101100" beta="101100"/>
          </csf>

Modified xml:

          <csf id="CSFcoeff_1" exctLvl="2" coeff="-0.14912" qchem_coeff="-0.14912" occ="220020">
            <det id="csf_1-0" coeff="1" alpha="110010" beta="110010"/>
            <det id="csf_1-0" coeff="1" alpha="101100" beta="101100"/>
          </csf>

This modification was applied to all degenerate CSFs, though only the first two are shown here for readability

Additional context Example energies at the cusp correction level:

• GAMESS reference: −230.8620665149
• QMC with unmodified XML: −230.847820 ± 0.014805
• QMC with modified XML (merged CSFs): −230.722973 ± 0.014256

[ye-luo]

It seems that the modified XML is already supported by QMCPACK. If this the case, the converter needs to fixed to keep the original symmetry.

During optimization, only the coeff of csf line should touched instead of individual det. Right? Do you know the behavior of the current code when running optimization using "Modified xml" style?

It is a bit concerning that both QMC energy doesn't reproduce the Gamess reference.

[NastaMauger]

Hello,

Are you implying that the current version of the code should be able to preserve the degeneracy of the coefficients? If so, how? I couldn’t find any option for this, and since the alpha and beta expansions are not the same—even though the coefficients are equal—it’s not surprising to have different energies with the modified XML (compared to GAMESS).

[NastaMauger]

Wouldn't it be cleaner and more mathematically rigorous to enforce that the derivative or Lagrangian of one CSF is equal to that of its degenerate counterpart?

That is what I aim to achieve in my current PR.

[ye-luo]

I guess your "modified xml" correspond to "One attempted workaround involved combining degenerate CSFs into a single CSF with multiple determinants. However, this manual modification led to noticeably different total energies and degraded the performance of cusp corrections, indicating that it is not a reliable solution." Should we figure out why it produces different energies? Are the energy values you mentioned from VMC without optimization? "degraded the performance of cusp corrections" also doesn't make sense to me. cusp correction is applied on the orbitals not affected by multi dets.

I'm looking for an xml input style that self-explaining the intended symmetry. Then we can enforce derivatives during optimization as you said. I feel the "modified xml" seems to be the right direction unless you disagree. If you agree, we just need to figure out why it doesn't produce intended energy and preserve symmetry during optimization.

Could you attached the full set of input files for the above two qmc runs you mentioned above?

[NastaMauger]

Hello,

Here are both outputs: cas_6_6_unmodified.wfj.txt cas_6_6_modified.wfj.txt

I have corrected my modified input, and the energy now looks reasonable: −230.822464 ± 0.01417.

However, I'm still unsure about how to handle CSFs with coefficients like 0.110793 and −0.110793. Even though I've grouped them under the same CSF, should their signs be preserved? And should the degeneracy still be enforced in this case?

How does the optimization procedure handle such situations?

[ye-luo]

Does this way work?

          <csf id="CSFcoeff_1" coeff="0.110793">
            <det id="csf_1-0" coeff="1.0" />
            <det id="csf_1-1" coeff="-1.0" />
          </csf>

I would expect the optimization only touching the coeff at csf not det

[NastaMauger]

Yes, it works. I have an energy of -230.847820 ± 0.014805, which is also very close to the GAMESS reference.

However, how are the individual coefficients (and hence their signs) taken into account in the optimization procedure with respect to the overall CSF group?

[NastaMauger]

My main concern with your current modification is whether having degenerate configurations within the same CSF group, with corresponding coefficient signs, is equivalent to having several degenerated CSF—but with a restricted optimization that preserves this degeneracy?

In other words, is this:

          <csf id="CSFcoeff_1" coeff="0.110793">
            <det id="csf_1-0" coeff="1.0" />
            <det id="csf_1-1" coeff="-1.0" />
          </csf>

equivalent to

          <csf id="CSFcoeff_0" coeff="0.110793">
            <det id="csf_0-0" coeff="1.0" />
          </csf>
          <csf id="CSFcoeff_1" coeff="-0.110793">
            <det id="csf_1-0" coeff="1.0" />
           </csf>

assuming the second case corresponds to a constrained optimization?

My naive guess is no, but my understanding and knowledge about the optimizer and how it is handled within QMCPACK is quite limited. Your point of view would be appreciated. Depending on this, I could update my PR accordingly

[ye-luo]

The overall coefficient of a determinant is the product of coeff on csf line and det https://github.com/QMCPACK/qmcpack/blob/a005bd7648639893b2e1559127206c1f42cdf175/src/QMCWaveFunctions/Fermion/MultiSlaterDetTableMethod.cpp#L787

here is how the derivatives dlogpsi of csf with respect to coeffs of csf computed. https://github.com/QMCPACK/qmcpack/blob/a005bd7648639893b2e1559127206c1f42cdf175/src/QMCWaveFunctions/Fermion/MultiSlaterDetTableMethod.cpp#L1058C1-L1058C19

In the following example

          <csf id="CSFcoeff_1" coeff="0.110793">
            <det id="csf_1-0" coeff="0.5" />
            <det id="csf_1-1" coeff="-0.5" />
          </csf>

The overall coefficients are 0.0553965 and -0.0553965. Then during optimization, 0.110793 is touched by the optimizer but 0.5 and -0.5 is frozen to preserve the degeneracy.

If breaking the degeneracy is intended, then following input can be used

          <csf id="CSFcoeff_0" coeff="0.110793">
            <det id="csf_1-0" coeff="0.5" />
          </csf>
          <csf id="CSFcoeff_1" coeff="0.110793">
            <det id="csf_1-0" coeff="-0.5" />
          </csf>

Then CSFcoeff_0 and CSFcoeff_1 are optimized freely without constraints.

Hypothetically if you have a capable optimizer which understand how to apply constraints CSFcoeff_0=CSFcoeff_1 in the case 2, then 1 and 2 can be made equivalent.

Currently in QMCPACK, you cannot express a constraint when using case 2 but case 1 does give the constraint you need I assume.