stylable

stylable copied to clipboard

cuda driver version is insufficient for cuda runtime version

11

I downloaded docker image, and then ran tensorflow according to step by step tutorial, but found an error. 

blackjack2016

Distributed training: Check failed: pb.ParseFromArray(meta_buf, buf_size) failed to parse string into protobuf, buf_size=172

5

**Describe the bug** I have a machine with 4 GPU's and I am trying to run a PyTorch based distributed job on all the GPU's. I instantiate 4 docker instances,...

ming9080

mxnet gluon example high CPU usage

3

**Describe the bug** ```sh python3 /usr/local/byteps/launcher/launch.py python3 /usr/local/byteps/example/mxnet/train_gluon_mnist_byteps.py ``` Training runs on GPU normally. It is werid that as soon as invoking `evaluate`, CPU usage skyrockets to ~87% (the worker...

jasperzhong

Shall we specify tags for the docker image, such as `bytepsimage/mxnet:v0.1` for byteps v0.1, and `bytepsimage/mxnet:master` for the master branch under development?

eric-haibin-lin

segmentation fault with mxnet built from source

7

``` root@dc4c46481a86:/usr/local/byteps/tests# TEST_TYPE=mxnet bash run_byteps_test.sh TEST MXNET ... BytePS launching worker warning: Error disabling address space randomization: Operation not permitted warning: Error disabling address space randomization: Operation not permitted warning:...

eric-haibin-lin

Suggestions for dockerfile - seperating dev and prod

1

Current dockerfiles are mainly for development, I suppose. Maybe for deployment cases, we can use [`nvidia/cuda:10.0-runtime-ubuntu18.04`](https://hub.docker.com/layers/nvidia/cuda/10.0-runtime-ubuntu18.04/images/sha256-099f29bff03c46491397a17c2caed4f958ec6d66957b91be01e1e0822a536136) as base container, and copy compiled files from development stage by leveraging[ multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/)....

jasperzhong

distributed training hang

29

I follow the [step-by-step-tutorial](https://github.com/bytedance/byteps/blob/master/docs/step-by-step-tutorial.md) to run distributed training with mxnet and tensorflow, both hang. I have 3 nodes and on first node I run scheduler and server and second and...

tingweiwu

tests/test_mxnet.py Failed

5

**Describe the bug** I am running the tests/test_mxnet.py in Amazon p2 instance, but the test failed. The output of "TEST_TYPE=mxnet bash tests/run_byteps_test.sh" is: ` TEST MXNET ... BytePS launching worker...

ChaokunChang

Performance slightly lower than horovod

3

**Describe the bug** We've been doing some benchmarking of horovod vs byte-ps on AWS. We were hoping to see some performance improvements from using byteps for 64 GPU jobs. We've...

bengineml

Bad Profile data

4

**Describe the bug** A clear and concise description of what the bug is. I use `BYTEPS_SERVER_KEY_TO_PROFILE` to output an `server_profile.json` file. But when load it using chrome, bugs like this...

xiongji