Feature Request: Add support for pytorch

[ludwigbald]

(This is a test issue. As mentioned in the responses to #3 and #4 , there is a development branch that supports pytorch. find it here: https://github.com/abahde/DeepOBS)

Expected behavior

I would appreciate if DeepOBS had built-in support for my pytorch optimizers.

This is relevant because a lot of optimizer research happens on pytorch.

Proposed approach:

Maybe @abahde could send a pull request when he's finished implementing it. Then @fsschneider can accept the pull request, handle the merging and we have success!

---

To Do

We will do the following steps for version 1.2.0:

• [ ] Add full support for PyTorch
  • [ ] Implement all Test Problems:
    • [ ] 2-D (data loading):
      • [ ] Beale
      • [ ] Branin
      • [ ] Rosenbrock
    • [x] Quadratic (data loading):
      • [x] Deep
    • [x] MNIST (data loading):
      • [x] Log. Regr.
      • [x] MLP
      • [x] 2c2d
      • [x] VAE
    • [x] Fashion-MNIST (data loading):
      • [x] Log. Regr.
      • [x] MLP
      • [x] 2c2d
      • [x] VAE
    • [x] CIFAR-10 (data loading):
      • [x] 3c3d
      • [x] VGG16
      • [x] VGG19
    • [x] CIFAR-100 (data loading):
      • [x] 3c3d
      • [x] VGG16
      • [x] VGG19
      • [x] All-CNN-C
      • [x] Wide ResNet-16-4
      • [x] Wide ResNet-40-4
    • [x] SVHN (data loading):
      • [x] 3c3d
      • [x] Wide ResNet-16-4
    • [ ] ImageNet (data loading):
      • [ ] VGG16
      • [ ] VGG19
      • [ ] Inception-v3
    • [x] Tolstoi (data loading):
      • [ ] CharRNN

[fsschneider]

We will do the following steps for version 1.2.0:

• [ ] Add full support for PyTorch

Added to the first post.

[abahde]

@fsschneider this list is correct at the time of merging my pull request. All ticks are correct.

[khizirsiddiqui]

Hey, is this still not resolved? and open for contributions?

[fsschneider]

Hey, it is indeed only partially solved on the development branch or the pre-release 1.2.0-beta. Most of the test problems are translated to PyTorch and are fully functioning (including the runner and all other necessities). But not all.

We are definitely open for contributions if you want to help us solve this issue.

[tvogels]

Hi! I'm looking forward to using this suite in PyTorch, and I'm happy to help a bit with the porting.

What's holding back the Tolstoi RNN task? I see there is both net_char_rnn and also a Tolstoi dataset, but no corresponding task.

Is there any particular challenge I am missing?

What is your procedure to test for equivalence between the TensorFlow and PyTorch implementations?

[fsschneider]

Hi @tvogels,

great to here it!

There is actually a PR that introduces the Tolstoi RNN task to PyTorch: #40.

I am currently working on a refactoring of DeepOBS for a new release. This will then include the open PRs, and a full PyTorch implementation. Unfortunately, my own dissertation has gotten in between my schedule. I don't have time to work on it in the next few weeks and can't offer a time estimation at the moment.

If you want to use DeepOBS for PyTorch, I would advise using the pre-release (or directly installing from the development branch). You can of course fork the entire repo to already use the Tolstoi PR if you wish.

[pratt3000]

Any updates? Will we have pytorch support anytime soon or has this become stale?

[fsschneider]

Hi @pratt3000,

the DeepOBS package is no longer maintained. I highly recommend using the AlgoPerf benchmark suite instead (I helped develop it). In my opinion, it is better in pretty much every aspect.

You can keep using DeepOBS, as is (we are also doing it for first prototypes). If you want to use PyTorch, I advise using the pre-release or directly installing it from the development branch. Although it doesn't support all aspects of the TensorFlow version of DeepOBS, it is usable. In our research, we use this version for initial experiments and use the AlgoPerf suite for more extensive experiments.