Displacement Memory Growth & Purge Option + Disk Caching of Displacements

[SoundsSerious]

Hi again,

I've been running a very large number of load combos in a distributed system, with an early stopping on failure routine. I am having trouble predicting the size of task to allocate by memory since the memory grows as I continue to perform further analysis on load combos.

I am currently addressing this with the following function running after I've processed the combo:

def purge_combo(struct:'Frame3D',combo:str):

    struct._D.pop(combo,None)
    for node in struct.Nodes.values():
        node.DX.pop(combo,None)
        node.DY.pop(combo,None)
        node.DZ.pop(combo,None)
        node.RX.pop(combo,None)
        node.RY.pop(combo,None)
        node.RZ.pop(combo,None)

This is working well and keeping my memory consumption per task consistent (and allowing me to fully utilize my AWS resources!)

A reorganization of some of the data structures to have one authoritative dictionary for displacements _D would be useful, and the node dictionaries DX,DY...RX could be replaced with weakref.WeakRefValueDictionary so that removal of the object from _D would automatically drop the reference from the node's as well without looping.

Of course this brings into question what happens if you want to reactivate the combo, and believe there might be some interesting options with https://github.com/grantjenks/python-diskcache although this invokes issues with how to identify the structure,combo pair.