Luciano Resende

@akchinSTC Could you please add to the description, the supported syntaxes you have in mind.

> @lresende After further testing and research, this approach will probably not work. There doesn't seem to be any support for "conditional entrypoints" in any packaging tools. We do have...

What would be a concrete example of a "bring your own generic component" that can't be exposed as either a script or a notebook? The issue is that runtimes are...

Also, generic components will have to reinvent the new KFP APIs, and we might go away from it.

As for the persistent layer, I would also recommend defining an abstraction layer so one can choose between mongo or any other data store they use/need.

Please make sure there is a good abstraction for the type of persistence, and if choosing MongoDB, also make sure the license is compatible. Looking at it today I saw:...

@saketanisha In case of YARN we actually used an existing python [YARN client](https://github.com/toidi/hadoop-yarn-api-python-client). If there isn't anything already available, and it's as simple as that, it can resides as a...

> Can someone help me understand how we IT k8 integration. For `mesos` integration I may need to spin single node `mesos` cluster inside docker ( kinda know the mechanics...

EMR is the Managed Spark from AWS, it seems to use YARN under the covers so our YarnClusterProcessProxy should work with any necessary tweaks to the environment configuration: https://aws.amazon.com/blogs/big-data/submitting-user-applications-with-spark-submit/

In principle, you are supposed to be able to perform the same Spark operation you can do on the application in the Scala kernel as well. One caveat is that...