presto-yarn
presto-yarn copied to clipboard
prestodb
Slider should not allow to allocate twice containers on the same node
As we do not support this we should prevent slider (yarn) somehow to not allocate two containers on the same node as it causes eventually whole presto cluster to fail with misleading errors. If it is not possible presto cluster should fail fast with a proper error message.
There are two scenarios which are vulnerable to this:
- COORDINATOR container should not be allocated on the node where WORKER container is allocated, and vice versa.
- when there is not enough nodes to allocate each container on separate node then presto-yarn should fail fast
When then number of requested nodes is equal, less than number of available nodes then first scenario is not an issue. So it looks like we need to first the second scenario.
According to https://slider.incubator.apache.org/docs/configuration/resources.html#placement-policy - Slider had planned a placement policy called anti affinity for this. Currently there's no support for anti-affinity —i.e. to mandate that component instances must never be deployed on the same hosts. Once YARN adds support for this, Slider will support it. Looks like https://issues.apache.org/jira/browse/SLIDER-82 is in progress
You are right, for the second scenario mentioned when there are not enough nodes to deploy number of Presto workers requested the application launch will fail. I will document this in the README while we come up with a fix for this.
It seems that https://issues.apache.org/jira/browse/SLIDER-82 is closed and should available in 0.91 slider version.