pytorch-operator

pytorch-operator copied to clipboard

Migrate code implementation to kubeflow/common fashion

3

kubeflow/common release a stable version 0.3.1 and we can migrate to use implementation of kubeflow/common. The change will be similar to change https://github.com/kubeflow/tf-operator/pull/1171. It would be better to resolve dependencies,...

Jeffwan

Do we need pod name and namespace in manifests?

2

https://github.com/kubeflow/pytorch-operator/blob/master/manifests/deployment.yaml#L22 I think we can remove it. /cc @johnugeorge WDYT, I think we do not use it

gaocegege

Why worker has init container wait for master ready?

3

 why not set large timeout at `torch.distributed.init_process_group(backend, init_method=None, timeout=datetime.timedelta(0, 1800), world_size=-1, rank=-1, store=None, group_name='')`? What's the meaning of adding this？

jiaqianjing

Cut release for pytorch operator

4

Tensorflow and PyTorch uses branches rather than tags for dependency management. Since we may make some breaking changes in the repo. I would suggest to cut a release and tags...

Jeffwan

[feature] Rethink distributed Pytorch backoff retry

6

Currently, the backoff retries in all replicas are controlled separately. Either directly restarted by pod controller (`restartPolicy` will be propagated from `ReplicaSpec` to `PodTemplateSpec`) https://github.com/kubeflow/pytorch-operator/blob/037cd1b18eb77f657f2a4bc8a8334f2a06324b57/pkg/controller.v1/pytorch/pod.go#L283-L289 or restarted in the pytorchjob...

czheng94

Example PytorchJob is not starting

8

I set up Kubeflow v0.6.0 on Microk8s v1.17. After executing kubectl create -f pytorch_job_mnist_gloo.yaml from the example I can see PytorchJob created, but no events on it happening and no...

natalytvinova

[examples/mnist]README.md instruction should be modified

1

The file directory changes from v1beta1 to v1. It needs to be fixed to v1 in README.md file ```bash kubectl create -f ./v1beta1/pytorch_job_mnist_gloo.yaml ```

sakaia

[examples/smoke_dist] pytorch_job_sendrecv.yaml does not exist in the directory

1

`examples/smoke-dist/README.md` documents `pytorch_job_sendrecv.yaml` But it does not exists on `examples/smoke-dist`

sakaia

kubeflow common dependency path to be updated

2

Taking [this fix](https://github.com/kubeflow/common/issues/54) into account, now pytorch operator should [depend on](https://github.com/kubeflow/pytorch-operator/blob/ec39dce0f98136ed89668c14130347616b463da5/pkg/apis/pytorch/v1/types.go#L18) `github.com/kubeflow/common/pkg/apis/common/v1` instead of `github.com/kubeflow/common/job_controller/api/v1`

igorvalko

cleanPodPolicy Set to Running should clean Running pod

5

https://github.com/kubeflow/pytorch-operator/blob/047cf0f41e68e030158f532017a226c18827a660/pkg/controller.v1/pytorch/job.go#L160 we just ignore running policy for now

xrmzju