Enhance UX for training

[ehsanmok]

Following up on the inference UX proposal, SageMaker requires many things to be changed and adjusted so that a normal working app / package becomes usable by SageMaker.

Pain-points

Essentially, a normal working Python package should be made to work with SageMaker without requiring things like entry_point to be in the root of the source_dir. We shouldn't require users to change the structure of their exising code base. Their package structure must be preserved. This makes adoption much easier.

For example,

package/
    notebook/
        train_deploy_sagemaker.ipynb
    src/
        foo
    tests/
    setup.cfg
    setup.py ---> might ignore notebook
    requirements.txt

train_deploy_sagemaker.ipynb to work requires change in the structure while pip install package followed by the notebook execution works in normal dev scenarios. This needs to be addressed.

Proposal

For better UX, try to minimize the module wrapping idea which makes debugging very hard. My proposal is decorators like

import sagemaker

@sagemaker.pytorch.entry_point(session, role, instance_count, instance_type, ...)
def foo(*args, **kwargs):
    ...

so that existing code base can become usable with minimal change. Hyperparameters can be passed to foo via cli.