Optimizations/memandgc2

[nostek]

the removedPorts change, I'm actually not sure it makes a difference as c# documentation says allocation doesn't happen until the first Add.

UnityEditor.TypeCache.GetTypesDerivedFrom() is Editor only, but speeds up reload time.

The remove of [serialized] is something I missed from the last PR.