Framework specification

[kszucs]

We should define a couple of frameworks which would implemented via mentor. A good starting point can be already implemented ones from here and here.

Write them in pure mentos (the core concepts are enough), then refactor to generalize the common parts.

There are a couple of domains we should cover:

1. Batch processing: implement a framework for dask distributed
2. Stream processing: implement a framework for rq
3. Serving: framework for rabbitmq
4. General task scheduling: like chronos, implement daskos

I think each of these have different needs.

[kszucs]

RQ

• RQ workers are identical, so we'll have homogeneous, long running tasks with same resource, environment needs.
• A task could be a shell command rq worker or just a pickled python function (I'm not sure which one is better - preference?)
• Due to no task-unique dependencies we have one and only mesos executor.
• To take advantage of bin-packing and shorten deployment time an executor can run multiple tasks simultaneously.
• We prioritize running a task in an existing executor over creating a new one (unless other constraints prohibit it).
• User has to be able to define framework role, and constraints like marathon/aurora enables it.

[Arttii]

I wanted to create a framework for Jupyter Notebooks as well,

Juyter

• Jupyter notebook to spin up a dynamic set of Jupyter Notebooks
• Each task is a single docker instance of Jupyter Notebook
• One non homogeneous task, which servers as an auth proxy to each Notebook
• Would most probably use docker overlay networking of mesos networking

For dask there was an example already create by Mrocklin I think no? We can reuse that I guess.

For RQ, do you mean we also spin up a redis instance and things or we completely remove redis from the picture and do what the current satyr implementation did for running arbitrary python functions somewhere?

There is also this: https://github.com/tobilg/mesos-framework/blob/master/examples/

[kszucs]

For RQ without redis. Mrockling had an example for running distributed on top of marathon which I've ported to daskathon. Can You implement these two as examples in mentos? Then we'll see what needs to be done in the pending PR.

[Arttii]

I think yes, I was a bit busy lately, but I will try to work on it a bit in the next few days

[kszucs]

I have another use case. Right now we're moving to https://clickhouse.yandex/ There are no mesos frameworks for it yet. How about implementing one (as a database example)?