population-irl icon indicating copy to clipboard operation
population-irl copied to clipboard

Speed up import time

Open AdamGleave opened this issue 7 years ago • 0 comments

It currently takes around 2 seconds to load the pirl module. Most of the time this isn't a big deal, but due to a combination of:

  1. We restart each Ray worker after a single call to work around TensorFlow/CUDA brokenness;
  2. A complete experiment can consist of 100s of calls; this adds up to a significant overhead when the results are mostly cached. (The overhead is still there for non-cached calls but is negligible compared to the cost of e.g. running an RL/IRL algorithm.)

I'm not sure how much we can realistically shave off. Just importing Ray takes ~0.5s, and we can't avoid that. Importing TensorFlow seems to be a big culprit; we actually can delay that, but we'd need to make imports in pirl.agents and pirl.irl be on-demand. (Currently they all get loaded by the config.)

AdamGleave avatar May 25 '18 00:05 AdamGleave