Upgrade pex in Heron

[billonahill]

Ideally we would fetch a pex binary at build time, but for now the code is committed and aging. This ticket is to upgrade pex to the latest release. In my tests this has worked fine, but we need to get this change merged into pex first: https://github.com/pantsbuild/pex/pull/277

[huijunwu]

Curious the motivation to choose PEX. why not use the Bazel py_binary, which could make the building process easier?

[joshfischer1108]

Following up from the conversation on Slack to keep others aware. I believe that at the time the core development in Heron was happening pi_binary rules did not exist. It would probably be easier to migrate to the pi_binary rules moving forward.

[nicknezis]

It seems there are two locations that are relevant to the PEX discussion. Are there any I'm missing? What changes would be needed to use the output of py_binary build task? Would this be easier or harder for analytics developers? I'm currently looking to use {Py}Gradle and the LinkedIn PEX plugin to package my analytics. It would not be ideal if we forced analytic developers to only use Bazel.

1. Heron Executor: Runs a pyton binary with the pex file as an argument (per component) https://github.com/apache/incubator-heron/blob/e1dda9acfded22ac1de86d73981118488d7a3b19/heron/executor/src/python/heron_executor.py#L724
2. Heron CLI checks for pex extension: https://github.com/apache/incubator-heron/blob/e1dda9acfded22ac1de86d73981118488d7a3b19/heron/tools/cli/src/python/submit.py#L439
3. Heron CLI: Uses pex_loader library to load the Topology class at submit time. https://github.com/apache/incubator-heron/blob/e1dda9acfded22ac1de86d73981118488d7a3b19/heron/tools/cli/src/python/execute.py#L124

[joshfischer1108]

Could you describe what an analytic developer is?

[nicknezis]

A developer writing a Heron topology to run in a managed instance of Heron.