learn2learn icon indicating copy to clipboard operation
learn2learn copied to clipboard

Add functorch improvements to inner-loop algorithms

Open DubiousCactus opened this issue 3 years ago • 1 comments

With PyTorch's latest update, the introduction of the functorch library allows, via composable function transforms, to "efficiently batching together tasks in the inner-loop of MAML". There seems to be a new way to vectorise gradient computation on a batch of tasks during the adaptation phase. I will try and look into this when I have some time, and maybe submit a PR. Unless someone knows how to do this? :)

DubiousCactus avatar Apr 22 '22 20:04 DubiousCactus

I'm not sure batching tasks will be very helpful because we're already running OOM when looping through individual tasks with MAML. I'll leave this open for now and we can see if there's more interest in the future.

seba-1511 avatar May 29 '23 01:05 seba-1511