OpenFF Gen 2 Torsion Set (v1.0) series

[hyejang]

For OpenFF Gen 2 Torsion Set series, so far there are 19 error jobs. Here I attach the list of error jobs for further fix.

TorsionDriveRecord(id='18045372', status='ERROR')
TorsionDriveRecord(id='18045425', status='ERROR')
TorsionDriveRecord(id='2703602', status='ERROR')
TorsionDriveRecord(id='18045499', status='ERROR')
TorsionDriveRecord(id='2703594', status='ERROR')
TorsionDriveRecord(id='18045715', status='ERROR')
TorsionDriveRecord(id='18045716', status='ERROR')
TorsionDriveRecord(id='18045852', status='ERROR')
TorsionDriveRecord(id='18045856', status='ERROR')
TorsionDriveRecord(id='18045859', status='ERROR')
TorsionDriveRecord(id='18045864', status='ERROR')
TorsionDriveRecord(id='18045865', status='ERROR')
TorsionDriveRecord(id='18045870', status='ERROR')
TorsionDriveRecord(id='18045875', status='ERROR')
TorsionDriveRecord(id='18045879', status='ERROR')
TorsionDriveRecord(id='18045886', status='ERROR')
TorsionDriveRecord(id='18045891', status='ERROR')
TorsionDriveRecord(id='18045896', status='ERROR')
TorsionDriveRecord(id='18045901', status='ERROR')
 total 19 errors found.

@dgasmith , could you help me with this issue? Thanks.

[dgasmith]

I have reset most of these. The OpenFF Gen 2 Torsion Set 1 Roche seems to have very high failures (~50%). Can you look through these and makes sure they look ok?

[hyejang]

Yes, it looks pretty unreasonable. I'll check the inputs for the set.

[dgasmith]

An update here for the Roche set, it looks like the larger molecules all failed while the smaller molecules completed fine. The logs do not have a full history here for some reason. We will be retrying them shortly with a higher memory limit from PRP.

[dotsdl]

All optimization tasks with state 'ERROR' have been restarted for the Gen 2 Torsion series of collections, both the first series and series 2. All torsiondrive services with state 'ERROR' have also been restarted.

We still see many cases of 'INCOMPLETE' optimization tasks. All of these appear to be coming from Lilac. @jchodera, can you verify that the manager(s) on Lilac have been shut down?