Instrumenting Data Analytics with Pin

[sarabjeetsingh007]

Hi, I was trying to attach Pin to instrument Data Analytics workload. My methodology is following:

1. I follow the guidelines from the guide page.
2. I install Pintool within the docker containers of master and slaves
3. All docker containers have privileged flag and can attach to a running process (kernel.yama.ptrace_scope=0). I have tested out a simple pintool with sample code.
4. Initiate the MapReduce workload by running master and slaves
5. Meanwhile, go inside one of the slave docker container. Look for the pid of working process. Attach pin to it.

My issue is that as soon as I attach pin to the process on slave machine, the process instantly terminates. Can't seem to understand why that it. I have attached the log that prints out on master as the slave crashes. Is there a simpler way to instrument/profile the benchmarks?

` ob: map 75% reduce 0% 20/09/25 08:04:59 INFO Job: Task Id : attempt_1601019528768_0018_m_000000_0, Status : FAILED [2020-09-25 08:04:58.737]Container [pid=3562,containerID=container_1601019528768_0018_01_000004] is running beyond virtual memory limits. Current usage: 619.7 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container. Dump of the process-tree for container_1601019528768_0018_01_000004 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 3566 3562 3562 3562 (java) 2594 50 4555460608 157983 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1848m -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1601019528768_0018/container_1601019528768_0018_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/hadoop-2.9.2/logs/userlogs/application_1601019528768_0018/container_1601019528768_0018_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 172.18.0.5 42741 attempt_1601019528768_0018_m_000000_0 4 |- 3562 3560 3562 3562 (bash) 0 0 3743744 668 /bin/bash -c /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1848m -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1601019528768_0018/container_1601019528768_0018_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/hadoop-2.9.2/logs/userlogs/application_1601019528768_0018/container_1601019528768_0018_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 172.18.0.5 42741 attempt_1601019528768_0018_m_000000_0 4 1>/opt/hadoop-2.9.2/logs/userlogs/application_1601019528768_0018/container_1601019528768_0018_01_000004/stdout 2>/opt/hadoop-2.9.2/logs/userlogs/application_1601019528768_0018/container_1601019528768_0018_01_000004/stderr

[2020-09-25 08:04:58.743]Container killed on request. Exit code is 143 [2020-09-25 08:04:58.753]Container exited with a non-zero exit code 143. `

I checked out the memory and disk usage of the containers. They seem to be well below the bounds.

Thanks, Sarab