Nic Eggert

Thank you!

@kimwnasptd Any movement on this? Our users run into this issue fairly regularly and it's not easy for someone who doesn't know Kubernetes to debug.

It's worth noting that there's a [related issue](https://github.com/kubeflow/kubeflow/issues/6886). If the string is left blank (or after this PR, all whitespace), it breaks the notebook dashboard entirely.

Ideally, the service should look like this (unsure about the gcs port): ``` ports: - appProtocol: grpc name: client port: 10001 protocol: TCP targetPort: 10001 - appProtocol: http name: dashboard...

I was able to figure out a hacky workaround for this. We can add port definitions using the `headService` field. However, these ports get _appended_ to what's in `headGroupSpec.template.spec.containers[0].ports`. If...

I've seen something similar when running on yarn. I didn't dig into it in detail, but the basic symptom is that the yarn job finishes fitting, but never exits.

I'm seeing a pretty significant performance hit on RedPajama-7b-chat that I think is due to this change. I ran the PyTorch profiler and all of the `repeat` operators in `apply_rotary_pos_emb`...

We just had a user run into this, so the issue is still present in Kubeflow 1.7.0.

Great, thanks for the quick fix. We'll give it a shot once the release drops.

@cdesiniotis This issue does seem to be resolved, but I'm still running into #399