Wei Chen

hi, how to involve this issue.

I see. I can take a try on this issue.

add 2 configurations to yarn-site.xml key: yarn.resourcemanager.docker-container-monitor.enabled; default value:false key:yarn.resourcemanager.docker-python-rpc-object; default value:container.master

hi, Xiandong Thanks for interesting, I wrote a generator here for my paper: https://github.com/yncxcw/YarnBench It now can (1) support trace jobs generation( from a trace file) and poisson job generation(...

Hi, Xiandong For Q1, I would rather say Big-c is a yarn on docker. Since yarn now already supports docker containers, we leverage the docker as well as cgroup in...

@xiandong79 I understand. I think streaming jobs may not be fit for this situation because the spark executors are allocated at the beginning of the job and the resources will...

hi Xiecheng Sorry for some miss-understanding, PyDockerMonitor is just my first version of this project, in which I implemented it as a standalone service and communicate with Yarn through RPC...

Hi, xiecheng That's fine. I am happy to discuss research questions. 1. For head-of-line blocking, it really exists in some production systems, since small jobs are often blocked by long...

hi, xiecheng That's OK. For 1, yes, the motivation of these projects is to minimize the queueing delays, either queued at master node, like yarn or queuing at slave node,...

That's OK~