byte-buddy icon indicating copy to clipboard operation
byte-buddy copied to clipboard

Byte-buddy J9 attachment fails in Rootless Podman Container due to Permission Denied

Open Jefiya-MJ opened this issue 7 months ago • 1 comments

Hi @raphw , We noticed that J9 attachment issue for one of the scenario where container is running as rootless podman and it fails due to a PermissionDenied error when it tries to access attach info via /proc filesystem. The failure specifically occurs during the attach mechanism that attempts to access the path /proc/<pid>/root/tmp/.com_ibm_tools_attach/_attachlock.
In the logs we say following error during attach:

2025-05-19T14:37:56.239+02:00 | ERROR | instana-executor-thread-2-2474   | AttachHelperImpl | com.instana.agent-jvm-handling - 1.0.17 | NOTE: Picked up JDK_JAVA_OPTIONS: 
java.io.FileNotFoundException: /proc/120/root/tmp/.com_ibm_tools_attach/_attachlock (Permission denied)
	at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:344)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:213)
	at net.bytebuddy.agent.VirtualMachine$ForOpenJ9.attach(VirtualMachine.java:1686)
	at net.bytebuddy.agent.VirtualMachine$ForOpenJ9.attach(VirtualMachine.java:1666)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:575)

This results in the failure of the VirtualMachine.java call for J9.

access to /proc/120/root/tmp/.com_ibm_tools_attach/_attachlock is denied: Inside the container:

[root@itew-ir-wlp-spoc01-dummy-app-wlp /]#ls -latr /proc/120/root/tmp/.com_ibm_tools_attach/_attachlock
ls: cannot access '/proc/120/root/tmp/.com_ibm_tools_attach/_attachlock': Permission denied
[root@itew-ir-wlp-spoc01-dummy-app-wlp /]# ls -latr /tmp/.com_ibm_tools_attach/_attachlock
-rw-rw-rw-. 1 default root 0 Mar 27 11:48 /tmp/.com_ibm_tools_attach/_attachlock

Rootless Podman containers are isolated using user namespaces and may also restrict /proc//root acess. Question: Would it be possible to bypass the attach mechanism’s dependency on /proc//root/tmp/... and instead attempt the attach directly via: /tmp/.com_ibm_tools_attach/_attachlock for the customer they can't try access via proc in rootless namespace. This would allow the attach to succeed in scenarios where /proc access is restricted due to rootless Podman but the attachfile file is otherwise accessible via /tmp. Is there a workaround or supported mechanism to allow attachment in such environments? Any guidance or suggestions on handling JVM attachment in rootless Podman containers — particularly when /proc/[pid]/root access is restricted — would be greatly appreciated. I am not sure getTemporaryFolder(processId) here even if /proc is not accessible why it is returning /proc/pid/root/tmp?

Jefiya-MJ avatar May 29 '25 05:05 Jefiya-MJ

You can already set -Dcom.ibm.tools.attach.directory=/tmp/.com_ibm_tools_attach which is the official mechanism in OpenJ9. Would this be an option to define in this container?

raphw avatar Jun 02 '25 20:06 raphw