Jonathan Hseu

Yes, unfortunately this is a known issue. There's no way to automatically stop parameter servers when the job is done at the moment. I'll ping back on this issue when...

I'm not opposed to this, but wouldn't it be better to wait until Spark 3.0.0 is released?

Yeah, I don't mind keeping this open.

Are you setting the train_dir to a local directory? It must be a directory visible to all workers.

Nope, but I think it makes sense to add support for that to TensorFlow, so contributions are welcome. It should be as a tf.data.Dataset. @mrry FYI

@skavulya Would you mind taking a look if you have time? @zhangxuhong Mind adding a test?

Thanks for the pull request! Mind handling the CLA before I take a look?

Hey @tslam75, thanks for the PR! Considering that this changes significantly with Hadoop 3.0, perhaps let's move it to another repository you own and add a link to the top-level...

Thanks! That'd be really useful.

Other languages external to Google aren't really handling it yet. You'll need to call the TF_LoadLibrary() C API function on the specific .so file that's included with the pip install...