TorsionDrive Integrity Error

[dgasmith]

Describe the bug The TorsionDrive id 2703253 is missing several geometry optimizations in its history lookup.

To Reproduce

td = client.query_procedures(id=2703253)[0]
print(len(td.optimization_history["[120]"]), td.minimum_positions["[120]"])
3, 3

Where if minimum_positions is 3 it must index a optimization_history of length 4 (index 3). This indicates that the database join table is missing several geometry optimizations. This causes several operations to fail when calling the TorsionDrive object.

Expected behavior minimum_positions must always index optimization_history.

Additional context It is unclear if this is a database error, a service class error, or other. Scanning all torsion drives, the only ones affected are:

for chunk in range(10):
    tds = client.query_procedures(procedure="torsiondrive", skip=chunk*1000)
    for td in tds:
        if td.status != "COMPLETE":
            continue
        if all(len(td.optimization_history[k]) > td.minimum_positions[k] for k in td.minimum_positions.keys()):
            continue
        
        print(td.id)

2703616
2703386
2703545
2703343
2703449
2703439
2703546
2703253

All of which were last modified July 14th, 2019. Importantly this is not an issue for the ~2500 newer torsion drives and only this set. So I am leaning a bit towards this being a migration bug or state bug rather than a current issue. Still this needs to be tracked down carefully.

[SimonBoothroyd]

We also came across this issue today with torsion drive id=6098580

[dotsdl]

@bennybp can you investigate where drops of optimization history may be happening? Does the source data exist in the database?

If it doesn't, this may be a case of the manager's submission to the server getting dropped and results not fully making it into the database. This is a problem downstream for fitting efforts, since the resulting optimizations composing the torsiondrive results may not actually give minimum energy results for each dihedral angle.

It may be that the work you're already doing to address non-atomic transactions from the manager -> server -> database will address this. If so, at some point we won't see new cases of this.

The failure mode identified by @SimonBoothroyd above is a loud case, but we have also identified some more silent cases that exhibit similar behavior, where the indices in the minimum_positions doesn't match what is actually present in optimization_history.

from qcportal import FractalClient
fc = FractalClient()
td = fc.query_procedures(id=6098580)[0]
td.get_final_results()   # IndexError

[bennybp]

I just wrote a quick script to test integrity, and there are some issues with the following ids. The most disturbing being the last, as it is only from last October (meaning something is still wrong somewhere, although much rarer).

The errors are not only for missing keys like this issue describes, but also missing optimization keys and mismatched energies (the energy in the final_energy does not match the energy from the stated minimum optimization record).

So now lets see if I can reconstruct what the data is supposed to be

Bad IDs:

1761773 1761604 1761727 1761737 1761628 1761645 1761808 1761885 1761881 1761961 1761953 1761865 1761911 1761838 1761912 1762696 1762338 1762219 1762634 1762771 1762377 2703253 2703343 2703365 2703386 2703439 2703449 2703451 2703545 2703546 2703616 6098501 6098580 24303714

[bennybp]

I had tried to fix these, but there are optimizations that I could not find, ad I don't fully trust the calculations anymore.

I have marked the 34 torsiondrive records mentioned above as invalid. They will still exist, but hopefully that serves as a signal that the results shouldn't be trusted. This status change is reversible if needed.

Nevertheless, I believe the bug that originally created this issue has been fixed.