oneflow Multi tensor optimizer

Features:

Add multi-tensor Python frontend for SGD, Adam, Adamw.
Modify unittest for multi-tensor cases for SGD, Adam, Adamw and delete the duplicated test_multi_tensor_* unittest files.
Modify clip_grad_norm_np for multi-tensor clip.
Support independent multi-tensor momentum kernel for SGD.

Performance tests: https://github.com/Oneflow-Inc/OneTeam/issues/1698#issuecomment-1312488683

Oct 18 '22 15:10 rejoicesyc

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

Oct 18 '22 16:10 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.4ms (= 13941.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 161.1ms (= 16111.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.16 (= 161.1ms / 139.4ms)

OneFlow resnet50 time: 84.8ms (= 8481.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 111.1ms (= 11113.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.31 (= 111.1ms / 84.8ms)

OneFlow resnet50 time: 57.4ms (= 11487.1ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.3ms (= 15656.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 78.3ms / 57.4ms)

OneFlow resnet50 time: 44.6ms (= 8923.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 74.7ms (= 14948.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.68 (= 74.7ms / 44.6ms)

OneFlow resnet50 time: 39.4ms (= 7870.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 68.4ms (= 13680.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.74 (= 68.4ms / 39.4ms)

Oct 18 '22 16:10 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Oct 18 '22 16:10 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.5ms (= 13946.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 167.7ms (= 16767.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 167.7ms / 139.5ms)

OneFlow resnet50 time: 85.9ms (= 8588.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 112.2ms (= 11223.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.31 (= 112.2ms / 85.9ms)

OneFlow resnet50 time: 58.3ms (= 11665.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 79.3ms (= 15858.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 79.3ms / 58.3ms)

OneFlow resnet50 time: 44.4ms (= 8871.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.6ms (= 13915.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.57 (= 69.6ms / 44.4ms)

OneFlow resnet50 time: 40.0ms (= 7997.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 77.8ms (= 15559.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.95 (= 77.8ms / 40.0ms)

Oct 24 '22 03:10 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Oct 24 '22 03:10 github-actions[bot]

需要在一些真实的模型上测一下 eager 下对收敛有没有影响，比如 resnet50 ,swin 等

Oct 25 '22 01:10 Ldpe2G

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

Oct 27 '22 17:10 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.9ms (= 13993.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 162.6ms (= 16255.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.16 (= 162.6ms / 139.9ms)

OneFlow resnet50 time: 86.6ms (= 8663.9ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 103.2ms (= 10321.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 103.2ms / 86.6ms)

OneFlow resnet50 time: 58.1ms (= 11627.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.3ms (= 15663.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 78.3ms / 58.1ms)

OneFlow resnet50 time: 45.5ms (= 9090.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.3ms (= 15653.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.72 (= 78.3ms / 45.5ms)

OneFlow resnet50 time: 41.0ms (= 8204.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.3ms (= 13467.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.64 (= 67.3ms / 41.0ms)

Oct 28 '22 02:10 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.7ms (= 13967.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 163.6ms (= 16360.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 163.6ms / 139.7ms)

OneFlow resnet50 time: 85.1ms (= 8506.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 101.8ms (= 10175.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 101.8ms / 85.1ms)

OneFlow resnet50 time: 57.4ms (= 11485.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.1ms (= 15614.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 78.1ms / 57.4ms)

OneFlow resnet50 time: 45.0ms (= 9006.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.1ms (= 13817.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.53 (= 69.1ms / 45.0ms)

OneFlow resnet50 time: 41.5ms (= 8309.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.4ms (= 13874.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.67 (= 69.4ms / 41.5ms)

Oct 31 '22 02:10 github-actions[bot]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

Nov 03 '22 03:11 github-actions[bot]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

Nov 03 '22 03:11 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.7ms (= 13965.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.2ms (= 16017.1ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 160.2ms / 139.7ms)

OneFlow resnet50 time: 84.6ms (= 8464.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 101.5ms (= 10146.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 101.5ms / 84.6ms)

OneFlow resnet50 time: 57.5ms (= 11508.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.2ms (= 15634.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 78.2ms / 57.5ms)

OneFlow resnet50 time: 44.4ms (= 8874.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 81.3ms (= 16265.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.83 (= 81.3ms / 44.4ms)

OneFlow resnet50 time: 40.2ms (= 8044.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.7ms (= 13538.1ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.68 (= 67.7ms / 40.2ms)

Nov 03 '22 06:11 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Nov 03 '22 06:11 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.5ms (= 13952.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.5ms (= 16054.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 160.5ms / 139.5ms)

OneFlow resnet50 time: 85.0ms (= 8500.7ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 112.0ms (= 11203.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 112.0ms / 85.0ms)

OneFlow resnet50 time: 57.7ms (= 11534.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 88.5ms (= 17691.4ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.53 (= 88.5ms / 57.7ms)

OneFlow resnet50 time: 44.5ms (= 8895.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 72.2ms (= 14434.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.62 (= 72.2ms / 44.5ms)

OneFlow resnet50 time: 40.0ms (= 8006.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 75.9ms (= 15187.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.90 (= 75.9ms / 40.0ms)

Nov 03 '22 09:11 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Nov 03 '22 09:11 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Nov 03 '22 10:11 github-actions[bot]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

Nov 03 '22 14:11 github-actions[bot]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

Nov 03 '22 14:11 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.4ms (= 13935.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 162.4ms (= 16242.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 162.4ms / 139.4ms)

OneFlow resnet50 time: 84.8ms (= 8475.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 111.6ms (= 11157.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 111.6ms / 84.8ms)

OneFlow resnet50 time: 57.7ms (= 11542.1ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 86.7ms (= 17341.4ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.50 (= 86.7ms / 57.7ms)

OneFlow resnet50 time: 44.1ms (= 8810.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.5ms (= 15701.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.78 (= 78.5ms / 44.1ms)

OneFlow resnet50 time: 40.1ms (= 8028.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.3ms (= 15668.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.95 (= 78.3ms / 40.1ms)

Nov 03 '22 15:11 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Nov 03 '22 15:11 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.5ms (= 13952.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.9ms (= 16094.1ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 160.9ms / 139.5ms)

OneFlow resnet50 time: 84.8ms (= 8482.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 109.2ms (= 10919.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.29 (= 109.2ms / 84.8ms)

OneFlow resnet50 time: 57.6ms (= 11512.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.1ms (= 15625.4ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 78.1ms / 57.6ms)

OneFlow resnet50 time: 44.8ms (= 8957.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.8ms (= 13958.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.56 (= 69.8ms / 44.8ms)

OneFlow resnet50 time: 40.2ms (= 8038.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.3ms (= 13461.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.67 (= 67.3ms / 40.2ms)

Nov 05 '22 12:11 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.4ms (= 13940.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.3ms (= 16030.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 160.3ms / 139.4ms)

OneFlow resnet50 time: 84.8ms (= 8484.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 100.9ms (= 10087.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 100.9ms / 84.8ms)

OneFlow resnet50 time: 57.7ms (= 11544.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 77.8ms (= 15565.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 77.8ms / 57.7ms)

OneFlow resnet50 time: 46.2ms (= 9248.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 75.8ms (= 15156.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.64 (= 75.8ms / 46.2ms)

OneFlow resnet50 time: 40.4ms (= 8078.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 68.8ms (= 13765.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.70 (= 68.8ms / 40.4ms)

Nov 05 '22 14:11 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Nov 05 '22 14:11 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 140.1ms (= 14007.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.5ms (= 16054.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 160.5ms / 140.1ms)

OneFlow resnet50 time: 85.2ms (= 8518.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 101.8ms (= 10176.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 101.8ms / 85.2ms)

OneFlow resnet50 time: 58.1ms (= 11617.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.4ms (= 15681.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 78.4ms / 58.1ms)

OneFlow resnet50 time: 44.9ms (= 8978.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 81.2ms (= 16232.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.81 (= 81.2ms / 44.9ms)

OneFlow resnet50 time: 41.5ms (= 8303.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.6ms (= 13126.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.58 (= 65.6ms / 41.5ms)

Nov 11 '22 03:11 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Nov 11 '22 03:11 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.9ms (= 13986.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 163.0ms (= 16295.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 163.0ms / 139.9ms)

OneFlow resnet50 time: 85.1ms (= 8513.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 101.1ms (= 10109.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 101.1ms / 85.1ms)

OneFlow resnet50 time: 57.3ms (= 11467.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.0ms (= 15606.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 78.0ms / 57.3ms)

OneFlow resnet50 time: 44.3ms (= 8856.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 72.2ms (= 14437.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.63 (= 72.2ms / 44.3ms)

OneFlow resnet50 time: 39.4ms (= 7870.1ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 76.3ms (= 15258.1ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.94 (= 76.3ms / 39.4ms)

Nov 14 '22 04:11 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Nov 14 '22 05:11 github-actions[bot]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

Nov 17 '22 03:11 github-actions[bot]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.6ms (= 13959.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 165.9ms (= 16586.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 165.9ms / 139.6ms)

OneFlow resnet50 time: 85.2ms (= 8522.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 106.7ms (= 10671.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.25 (= 106.7ms / 85.2ms)

OneFlow resnet50 time: 57.9ms (= 11586.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.8ms (= 15769.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 78.8ms / 57.9ms)

OneFlow resnet50 time: 45.3ms (= 9069.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.1ms (= 13430.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.48 (= 67.1ms / 45.3ms)

OneFlow resnet50 time: 41.2ms (= 8248.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 80.7ms (= 16135.1ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.96 (= 80.7ms / 41.2ms)

Nov 17 '22 05:11 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9267/

Nov 17 '22 05:11 github-actions[bot]

oneflow oneflow copied to clipboard

Multi tensor optimizer

oneflow
oneflow copied to clipboard