Haibin Lin

Extend bert/embedding.py with other pre-trained models

2

We shall extend bert/embedding.py with other pre-trained models like XLNet, RoBERTa, R-XLM, ALBERT, etc. This is a good starting script where people try to leverage pre-trained model embeddings for their...

bit mask support for dropout

4

For dropout training, one can save the dropout mask with 1 bit per coordinate. Can we support that in DNNL? Memory is precious.

MacOS cmake build instruction broken

4

Following the instructions in https://github.com/apache/incubator-mxnet/blob/master/config/darwin.cmake: ``` # If you want to change the configuration, please use the following steps. # Assume you are on the root directory of mxnet. First...

Shall we specify tags for the docker image, such as `bytepsimage/mxnet:v0.1` for byteps v0.1, and `bytepsimage/mxnet:master` for the master branch under development?

segmentation fault with mxnet built from source

7

``` root@dc4c46481a86:/usr/local/byteps/tests# TEST_TYPE=mxnet bash run_byteps_test.sh TEST MXNET ... BytePS launching worker warning: Error disabling address space randomization: Operation not permitted warning: Error disabling address space randomization: Operation not permitted warning:...

Error at `byteps.shutdown`

3

**Describe the bug** test.py ``` import byteps.torch; byteps.torch.init(); import time; time.sleep(10); byteps.torch.shutdown(); ``` ``` export NVIDIA_VISIBLE_DEVICES=0; export DMLC_NUM_WORKER=1; export DMLC_WORKER_ID=0; export DMLC_ROLE=worker; export BYTEPS_LOG_LEVEL=DEBUG; python3 /usr/local/byteps/launcher/launch.py python3 test.py ``` Error...

Suggestions for documentation

1

It would be great to improve the documentation for first-time users. Current documentation assumes users know horovod APIs, but for users who just get into distributed training they do not...

Redirect stderr and stdout with byteps launcher

1

I updated the example training script with some intended error: ``` sym = net.get_symbol(**vars(args)) # train import adfsadsfdasf # err_log BytePS launching worker [2019-10-23 05:37:23.615065: D byteps/common/communicator.cc:63] Using Communicator=Socket [2019-10-23...

**Describe the bug** I forgot to use `bps.init()` at the beginning of the program, and got a long stacktrace. It was due to incorrect byteps usage, but I hope there...

fix: fix a typo

4

## Category: ## Description: ## Additional information: ### Affected modules and functionalities: ### Key points relevant for the review: ### Tests: - [x] Existing tests apply - [ ] New...