oneflow icon indicating copy to clipboard operation
oneflow copied to clipboard

Thread local vm need sync

Open lixinqi opened this issue 2 years ago • 6 comments

解决芯片算子api内部的fork死锁问题。

昇腾算子api的内部会起多线程,在每个线程里有可能调用fork,这是目前Master没法处理的情况。根本原因是atfork时候依赖了SyncVmMode,这是一个必须手工设置的变量,昇腾算子显然不知道在新建线程的时候需要设置SyncVmMode::kEnable,这才导致了死锁。

本pr让atfork不再依赖静态的SyncVmMode,而是依赖动态的boolean类型的VmNeedSync标记,这个标记会在VirtualMachine::Receive函数内实际发指令到vm的时刻才会设置。如果当前线程从未发过vm的指令,显然是不需要同步的。

lixinqi avatar Oct 27 '22 04:10 lixinqi

Speed stats:
GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.7ms (= 13972.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 163.4ms (= 16336.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 163.4ms / 139.7ms)

OneFlow resnet50 time: 86.4ms (= 8638.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 104.5ms (= 10451.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.21 (= 104.5ms / 86.4ms)

OneFlow resnet50 time: 59.3ms (= 11852.4ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.1ms (= 15615.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 78.1ms / 59.3ms)

OneFlow resnet50 time: 46.1ms (= 9222.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.2ms (= 13847.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.50 (= 69.2ms / 46.1ms)

OneFlow resnet50 time: 40.4ms (= 8079.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 68.9ms (= 13774.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.70 (= 68.9ms / 40.4ms)

github-actions[bot] avatar Oct 28 '22 06:10 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9319/

github-actions[bot] avatar Oct 28 '22 07:10 github-actions[bot]

Speed stats:

github-actions[bot] avatar Oct 31 '22 17:10 github-actions[bot]

Speed stats:
GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.7ms (= 13974.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.6ms (= 16059.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 160.6ms / 139.7ms)

OneFlow resnet50 time: 85.2ms (= 8518.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 101.7ms (= 10170.9ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 101.7ms / 85.2ms)

OneFlow resnet50 time: 57.4ms (= 11475.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 77.9ms (= 15571.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 77.9ms / 57.4ms)

OneFlow resnet50 time: 44.2ms (= 8841.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 71.1ms (= 14229.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.61 (= 71.1ms / 44.2ms)

OneFlow resnet50 time: 40.4ms (= 8074.1ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 73.1ms (= 14613.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.81 (= 73.1ms / 40.4ms)

github-actions[bot] avatar Oct 31 '22 20:10 github-actions[bot]

CI failed when running job: cuda-misc. PR label automerge has been removed

github-actions[bot] avatar Oct 31 '22 21:10 github-actions[bot]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9319/

github-actions[bot] avatar Oct 31 '22 21:10 github-actions[bot]