drill-env.sh configs doesn't seem to be in effect.

[wschung1113]

Hi.

First off I want to thank you for such contribution. It's hard to find Drill on k8s guidelines.

So I have deployed your helm chart on my on-prem cluster consisted of 1 master node and 3 slave nodes.

2 of the slave nodes have sufficient memory (32gbs) so I have set nodeAffinity to deploy drillbit pods only on those two nodes.

Our main purpose of using Drill is as a SQL query engine for RDBMS (Oracle alike) for a BI tool of our own.

I have set the memory values in values.yaml drill.memory as 13Gi and left cpu as default (4000m).

Reason being, I read thru the docs of Apache Drill and set drill-env.sh as:

export DRILL_PID_DIR="/opt/drill" export DRILLBIT_MAX_PROC_MEM=13G export DRILL_HEAP=4G export DRILL_MAX_DIRECT_MEMORY=8G

However, my question is that there is no difference to the performance of a query on a large data set (size of 27 Gi).

When everything was default and I haven't set up custom drill-env.sh, and drill.memory in values.yaml was only 5Gi, the query took a little over 4 minutes and it still takes the same time.

When I look at 'Metrics' tab it looks like this when running the query:

Also here is the node status with the drillbit pod running a query. Memory request does not change even after start running my SQL query.

So I am guessing drill-env.sh configurations doesn't seem to be in effect. (HEAP_MEM, MAX_DIRECT_MEM, ..etc).

Also, for our cluster set up, is 3 zookeeper pods enough, one for each slave node?

I have experienced that, when there was only one zookeeper pod deployed, multithreading of query execution was not possible. As in, query execution had to wait until query in progress was complete. However, after I increased the replicas of zookeeper pods in the statefulset, I could run two queries concurrently.

I am also curious if one drillbit pod takes care of one query all by itself, even the load of the query is large? or does the drillbit pods split up the workload in parallel?

Lastly, I am curious about the rule-of-thumb to set how many drillbit and zookeeper pods. For example in our cluster each of the slave nodes has specs of 1: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz 6 Cores 12 Threads 32Gib RAM

2: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz 4 Cores 4 Threads 8Gib RAM

3: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz 6 Cores 12 Threads 32Gib RAM

What should be the pod memory to CPU ratio for drillbit pods?

Thanks so much for reading!!