peloton
peloton copied to clipboard
Published
20 hours ago •
uber
uber
[Resource manager] Invalid memory address or nil pointer dereference
Operation:
- run up minicluster with command:
PELOTON=app make minicluster
- check container status
➜ peloton git:(master) PELOTON=app make minicluster
PELOTON=app ./scripts/minicluster.sh setup
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
tools/minicluster/main.py:61: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(f)
started container peloton-cassandra
failed to create cassandra store, retrying...
failed to create cassandra store, retrying...
cassandra store is created
started container peloton-zk
sleep 20 secs for zk to come up
started container peloton-mesos-master
started container peloton-mesos-agent0
started container peloton-mesos-agent1
started container peloton-mesos-agent2
docker image "uber/peloton" has to be built first locally by running IMAGE=uber/peloton make docker
app peloton-hostmgr0 is not up yet, retrying...
app peloton-hostmgr0 is not up yet, retrying...
app peloton-hostmgr0 is not up yet, retrying...
started peloton-hostmgr0
app peloton-resmgr0 is not up yet, retrying...
started peloton-resmgr0
app peloton-placement0 is not up yet, retrying...
started peloton-placement0
app peloton-placement1 is not up yet, retrying...
started peloton-placement1
app peloton-jobmgr0 is not up yet, retrying...
started peloton-jobmgr0
app peloton-aurorabridge0 is not up yet, retrying...
started peloton-aurorabridge0
➜ peloton git:(master) docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9409deaf5e49 uber/peloton "/bin/entrypoint.sh" 25 seconds ago Up 24 seconds 0.0.0.0:5396->5396/tcp, 5290-5293/tcp, 0.0.0.0:8282->8282/tcp peloton-aurorabridge0
dfa10c61cfeb uber/peloton "/bin/entrypoint.sh" 30 seconds ago Up 29 seconds 5290-5291/tcp, 0.0.0.0:5292->5292/tcp, 5293/tcp, 0.0.0.0:5392->5392/tcp peloton-jobmgr0
f07c49345fc4 uber/peloton "/bin/entrypoint.sh" 36 seconds ago Up 35 seconds 0.0.0.0:5303->5303/tcp, 5290-5293/tcp, 0.0.0.0:5403->5403/tcp peloton-placement1
b0962774e39c uber/peloton "/bin/entrypoint.sh" 42 seconds ago Up 41 seconds 0.0.0.0:5293->5293/tcp, 5290-5292/tcp, 0.0.0.0:5393->5393/tcp peloton-placement0
7c4ab56fe895 uber/peloton "/bin/entrypoint.sh" 48 seconds ago Up 46 seconds 0.0.0.0:5290->5290/tcp, 0.0.0.0:5394->5394/tcp, 5291-5293/tcp peloton-resmgr0
ddffb662e05b uber/peloton "/bin/entrypoint.sh" About a minute ago Up About a minute 5290/tcp, 0.0.0.0:5291->5291/tcp, 5292-5293/tcp, 0.0.0.0:5391->5391/tcp peloton-hostmgr0
b312e9107c1f mesosphere/mesos-slave:1.6.0-rc1 "bash /files/run_mes…" About a minute ago Up About a minute 0.0.0.0:5053->5051/tcp peloton-mesos-agent2
c84a01d652f7 mesosphere/mesos-slave:1.6.0-rc1 "bash /files/run_mes…" About a minute ago Up About a minute 0.0.0.0:5052->5051/tcp peloton-mesos-agent1
6f321ccfb488 mesosphere/mesos-slave:1.6.0-rc1 "bash /files/run_mes…" About a minute ago Up About a minute 0.0.0.0:5051->5051/tcp peloton-mesos-agent0
354ae443d9e5 mesosphere/mesos-master:1.6.0-rc1 "bash /files/run_mes…" About a minute ago Up About a minute 0.0.0.0:5050->5050/tcp peloton-mesos-master
177c941daeb4 zookeeper:3.4 "/docker-entrypoint.…" About a minute ago Up About a minute 2888/tcp, 3888/tcp, 0.0.0.0:8192->2181/tcp peloton-zk
1ca9583ad784 cassandra:3.0.14 "bash /files/run_cas…" About a minute ago Up About a minute 7000-7001/tcp, 0.0.0.0:9042->9042/tcp, 7199/tcp, 0.0.0.0:9160->9160/tcp peloton-cassandra
- create resource pool according to [tutorial] with yaml (https://github.com/uber/peloton/blob/master/docs/tutorial.md)
name: HelloWorldPool
owningteam: MyTeam
ldapgroups:
- MyGroup
description: "My first resource pool on Peloton"
resources:
- kind: cpu
reservation: 3
limit: 6
share: 1
- kind: memory
reservation: 2048
limit: 4096
share: 1
- resource manager container exited with logs Related logs:
{"level":"info","msg":"LookupResourcePoolID called","request":{"path":{"value":"/"}},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"CreateResourcePool called","request":{"config":{"name":"HelloWorldPool","owningTeam":"MyTeam","ldapGroups":["MyGroup"],"description":"My first resource pool on Peloton","resources":[{"kind":"cpu","reservation":3,"limit":6,"share":1},{"kind":"memory","reservation":2048,"limit":4096,"share":1}],"parent":{"value":"root"}}},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Scheduling policy is not set, use default PriorityFIFO","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"siblings to check","name":"HelloWorldPool","siblingNames":{},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"a8006272-9181-4529-b4b0-feccf339f654","slack_limit":{"CPU":0,"MEMORY":409.6,"DISK":0,"GPU":0},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":3,"MEMORY":2048,"DISK":0,"GPU":0},"respool_id":"a8006272-9181-4529-b4b0-feccf339f654","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-06-18T08:39:35Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-06-18T08:39:35Z"}
{"demand_cap_by_limit":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_cap_by_reservation_assignment":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_not_satisfied":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"First pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"slack_entitlement":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-06-18T08:39:35Z","total_share":{}}
{"final_assignment":{"CPU":6,"MEMORY":0,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"time":"2019-06-18T08:39:35Z"}
{"final_assignment":{"CPU":6,"MEMORY":4096,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"time":"2019-06-18T08:39:35Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xbac011]
goroutine 611 [running]:
github.com/uber/peloton/resmgr/entitlement.(*Calculator).getChildShare(0xc000683b60, 0x1011aa0, 0xc0005f2360, 0xe90cf3, 0x4, 0x3ff0000000000000)
/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:182 +0xd1
github.com/uber/peloton/resmgr/entitlement.(*Calculator).distributeUnclaimedResources(0xc000683b60, 0x1011aa0, 0xc0005f2360, 0xc001cc38e8, 0xc001cc3a30)
/go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:320 +0x189
github.com/uber/peloton/resmgr/entitlement.(*Calculator).setEntitlementForChildren(0xc000683b60, 0x1011aa0, 0xc0005f2360)
/go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:75 +0x572
github.com/uber/peloton/resmgr/entitlement.(*Calculator).calculateEntitlement(0xc000683b60, 0x1002b00, 0xc0000b8070, 0x0, 0x0)
/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:162 +0x20e
github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start.func1(0xc000683b60, 0xc0004821c0)
/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:106 +0x138
created by github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start
/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:96 +0x133
Currently, I'm unable to reproduce this. What does your setup look like (OS, go version, python version)?
@TommyLike Can you pull the latest version and see if you're still reproducing this issue? I checked with our team and it looks like there was a fix put in for this, but would like to get some verification.
@TommyLike Just wanted to follow up to see if you were able to confirm that this issue has been resolved?
Hey @talaniz , I paste some of the configurations and logs here in case needed.
Environment:：Ubuntu 16.04 Docker version:
Client:
Version: 18.09.7
API version: 1.39
Go version: go1.10.8
Git commit: 2d0083d
Built: Thu Jun 27 17:56:17 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.7
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 2d0083d
Built: Thu Jun 27 17:23:02 2019
OS/Arch: linux/amd64
Experimental: false
Python Version:
Python 2.7.12
Images information:
uber/peloton latest sha256:36e4d089ba273cd6c3c88ec213e80a37b88aac16308c9bb12e3f017967886188 4c1d9f5761be 3 months ago 1.8GB
mesosphere/mesos-slave 1.6.0-rc1 sha256:b44439db7b5bb2e7742d8e83efaa82d6b276ab7287f0d99107ef6bdb6502599b 69562200cd36 14 months ago 1.02GB
mesosphere/mesos-master 1.6.0-rc1 sha256:e63cfdbd7265ad4a73c4e01bb8ff1393c3fcf49c68ce330b0c56fd3977f63d9f b2bbd52d0b2b 14 months ago 1.02GB
cassandra 3.0.14 sha256:616454e966db82cbea0fdf0b579fa827445de952214edbab5a23b63bc92a5aa0 3bdcdfbd60a0 21 months ago 380MB
Node info:
⚡ root@husheng-test ~/go_projects/src/github.com/uber/peloton/bin master ./peloton hostmgr hosts
Hostname| CPU| GPU| MEM| Disk| State| Task Hold
peloton-mesos-agent0| 4.00| 0.00| 2048.00 MB| 20000.00 MB| ready|
peloton-mesos-agent1| 4.00| 0.00| 2048.00 MB| 20000.00 MB| ready|
peloton-mesos-agent2| 4.00| 0.00| 2048.00 MB| 20000.00 MB| ready|
Resource pool yaml:
name: HelloWorldPool
owningteam: MyTeam
ldapgroups:
- MyGroup
description: "My first resource pool on Peloton"
resources:
- kind: cpu
reservation: 1
limit: 2
share: 1
- kind: memory
reservation: 1024
limit: 2048
share: 1
Parts of the log out put
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"GetActiveTasks called","req":{"states":["LAUNCHING","PENDING","PLACED","PLACING","PREEMPTING","READY"]},"time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"GetActiveTasks returned","time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:05Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:06Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:10Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:12Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:15Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:16Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:20Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:22Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:25Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:26Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:30Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:32Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:35Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"LookupResourcePoolID called","request":{"path":{"value":"/"}},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"CreateResourcePool called","request":{"config":{"name":"HelloWorldPool","owningTeam":"MyTeam","ldapGroups":["MyGroup"],"description":"My first resource pool on Peloton","resources":[{"kind":"cpu","reservation":1,"limit":2,"share":1},{"kind":"memory","reservation":1024,"limit":2048,"share":1}],"parent":{"value":"root"}}},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Scheduling policy is not set, use default PriorityFIFO","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"siblings to check","name":"HelloWorldPool","siblingNames":{},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"41b5f815-89ec-4a59-906b-c0623b2728c7","slack_limit":{"CPU":0,"MEMORY":204.8,"DISK":0,"GPU":0},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":1,"MEMORY":1024,"DISK":0,"GPU":0},"respool_id":"41b5f815-89ec-4a59-906b-c0623b2728c7","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:36Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:36Z"}
{"demand_cap_by_limit":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_cap_by_reservation_assignment":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_not_satisfied":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"First pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"slack_entitlement":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:36Z","total_share":{}}
{"final_assignment":{"CPU":2,"MEMORY":0,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"time":"2019-07-04T09:41:36Z"}
{"final_assignment":{"CPU":2,"MEMORY":2048,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"time":"2019-07-04T09:41:36Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xbac011]
goroutine 79 [running]:
github.com/uber/peloton/resmgr/entitlement.(*Calculator).getChildShare(0xc00067fa40, 0x1011aa0, 0xc001632480, 0xe90cf3, 0x4, 0x3ff0000000000000)
/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:182 +0xd1
github.com/uber/peloton/resmgr/entitlement.(*Calculator).distributeUnclaimedResources(0xc00067fa40, 0x1011aa0, 0xc001632480, 0xc001ddb8e8, 0xc001ddba30)
/go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:320 +0x189
github.com/uber/peloton/resmgr/entitlement.(*Calculator).setEntitlementForChildren(0xc00067fa40, 0x1011aa0, 0xc001632480)
/go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:75 +0x572
github.com/uber/peloton/resmgr/entitlement.(*Calculator).calculateEntitlement(0xc00067fa40, 0x1002b00, 0xc0000400f0, 0x0, 0x0)
/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:162 +0x20e
github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start.func1(0xc00067fa40, 0xc0005584d0)
/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:106 +0x138
created by github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start
/go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:96 +0x133
@TommyLike thanks for this and apologies for the late reply. I'll check this with the team.