[Discussion] Implementation of entry file for PAI

[lhw362950217]

About how to call entry file in PAI.

Background On submitting job to PAI, we need to pack some resource into a tarball, then push the tarball to PAI, finally call an entry file in the resource to start the training/predicting and etc. Before, we generate the entry file in Go, all entry code is generated in a dedicated Python file. Now, we are migrate Go code to Python, we should do similar thing, however, in Python language this time.

Implementation proposals We have two ways:

1. Generate equivalent code like in Go.
2. As we are already in Python, we can pickle all needed params to a params.pkl, and write a fixed dispatcher logic in our codebase (not generated this time). In the fixed logic, we unpickle the params and call corresponding train/predict, etc, like implemented in here.

[typhoonzero]

We may have below runtime entries to submit training/predicting jobs:

runtime.pai.train(datasource, ....)
runtime.local.train(datasource, ...)
runtime.elasticdl.train(datasource, ...)

For the PAI runtime API, when we call runtime.pai.train(), we should generate a file entry.py，then entry.py calls runtime.local.train to execute the training on the cluster.

[Yancey1989]

In my opinion, to generate the entry.py is easier to debug, agree with op1.

[lhw362950217]

In my opinion, to generate the entry.py is easier to debug, agree with op1.

To generate an entry.py, we need a couple of code template, also upstream stage like feature derivation should output code snippet, not Python objects, which I'm not sure is simple. As for debugging, we can dump the pickled params to json, or just run the entry.py to load and print the params.