arthas icon indicating copy to clipboard operation
arthas copied to clipboard

Published 20 hours ago • verified publisher icon alibaba

查看elasticsearch运行状况报错

Open dangran opened this issue 6 years ago • 17 comments

  • [ ] I have searched the issues of this repository and believe that this is not a duplicate.

Environment

  • Arthas version: 3.0.4
  • Operating System version: mac 10.11.6
  • Java version of target JVM: 1.8
  • Java version of JVM used to attach: 1.8

Steps to reproduce this issue

  1. 启动es
  2. 启动arthas
  3. 输入es对应jvm序号

Expected Result

进入arthas对es运行状况查看

Actual Result

报错

If there is an exception, please attach the exception trace:

Calculating attach execution time...
Attaching to 2062 using version 3.0.4...
Start arthas failed, exception stack trace: 
com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize
	at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:121)
	at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:84)
	at com.taobao.arthas.core.Arthas.<init>(Arthas.java:25)
	at com.taobao.arthas.core.Arthas.main(Arthas.java:96)

dangran avatar Sep 25 '18 02:09 dangran

Hi, please check the standard output or standard error to see if there is any exceptions.

If my memory is not wrong, elastic search enables security manager that will prevent Arthas to attach.

We have to add the specific permission to Arthas to make it work.

ralf0131 avatar Sep 25 '18 22:09 ralf0131

在开启SecurityManager的情况下，运行Arthas，需要的权限如下，新建一个policy文件，比如/path/to/java.policy,内容如下：

$cat elasticdb/config/java.policy
grant codeBase "file:${user.home}/.arthas/lib/-" {
    permission java.io.FilePermission "${user.home}/.arthas/-", "read,write";
    permission java.io.FilePermission "${user.home}/logs/arthas/-", "read,write,delete";
    permission java.lang.RuntimePermission "createClassLoader";
    permission java.lang.RuntimePermission "getClassLoader";
    permission java.lang.RuntimePermission "modifyThreadGroup";
    permission java.lang.RuntimePermission "modifyThread";
    permission java.lang.RuntimePermission "shutdownHooks";
    permission java.lang.RuntimePermission "accessClassInPackage.sun.reflect";
    permission java.lang.RuntimePermission "accessClassInPackage.sun.net.www.protocol.http";
    permission java.lang.RuntimePermission "accessClassInPackage.sun.net.www.http";
    permission java.net.SocketPermission "127.0.0.1:3658", "listen,resolve";
    permission java.net.SocketPermission "127.0.0.1:8563", "listen,resolve";
    permission java.net.SocketPermission "*", "accept";
};

注意："file:${user.home}/.arthas/lib/-"这里最后一个横线(dash)表示lib目录下所有class文件和jar包，并适用于所有子目录

然后需要应用启动的时候添加-D参数-Djava.security.policy=file:///path/to/java.policy

如果是tomcat的应用，只需要修改${catalina.base}/conf/catalina.policy把上面的内容加进去即可。

ralf0131 avatar Sep 25 '18 22:09 ralf0131

今天在elasticsearch 6.3.2版本上试了下（jdk 1.8 u92），调了很长时间都没通过，各种报java.security.AccessControlException，建议维护一个最新的policy文件。

dixingxing0 avatar Jun 25 '19 04:06 dixingxing0

因为ES的权限限制很严，arthas本身需要的权限不是很清楚。所以目前只能建议ES去掉权限配置。

后续把arthas需要的权限都理清之后，再尝试给出一个配置。

hengyunabc avatar Jun 28 '19 04:06 hengyunabc

@dijingran 我用的6.3.2 也报错，es 日志报错如下：

java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/gary/logs/arthas" "read")
        at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
        at java.base/java.security.AccessController.checkPermission(AccessController.java:1042)
        at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
        at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
        at java.base/java.io.File.exists(File.java:815)
        at com.taobao.arthas.agent.AgentBootstrap.<clinit>(AgentBootstrap.java:40)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
Arthas server agent start...
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
java.lang.ExceptionInInitializerError
        at com.taobao.arthas.core.server.ArthasBootstrap.<clinit>(ArthasBootstrap.java:33)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at com.taobao.arthas.agent.AgentBootstrap.bind(AgentBootstrap.java:175)
        at com.taobao.arthas.agent.AgentBootstrap.access$000(AgentBootstrap.java:16)
        at com.taobao.arthas.agent.AgentBootstrap$1.run(AgentBootstrap.java:140)
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/gary/logs" "read")
        at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
        at java.base/java.security.AccessController.checkPermission(AccessController.java:1042)
        at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
        at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
        at java.base/java.io.File.exists(File.java:815)
        at com.taobao.arthas.core.util.LogUtil.detectArthasLogDirectory(LogUtil.java:86)
        at com.taobao.arthas.core.util.LogUtil.<clinit>(LogUtil.java:47)
        ... 8 more

我自己尝试了多次，搞了一天没弄好，实在没办法了，来反馈问题，arthas 官方能否解决一下？Elasticsearch现在用的人特别多，如果能解决这个问题，很多人都能用上。

garyelephant avatar Jun 28 '19 05:06 garyelephant

遇到同样的问题，请 @hengyunabc 抽时间看一看。

[ERROR] Start arthas failed, exception stack trace: 
com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize
	at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:121)
	at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:94)
	at com.taobao.arthas.core.Arthas.<init>(Arthas.java:28)
	at com.taobao.arthas.core.Arthas.main(Arthas.java:113)
[ERROR] attach fail, targetPid: 65948

tianmingxing avatar Jul 06 '19 07:07 tianmingxing

我在es 5.4.3下面是可以的 java.policy 内容如下：

/*
 * Licensed to Elasticsearch under one or more contributor
 * license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright
 * ownership. Elasticsearch licenses this file to you under
 * the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

grant {

    permission java.security.AllPermission;
    permission java.lang.RuntimePermission "*";
    permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
    permission java.net.NetPermission "*";
    permission java.net.SocketPermission "localhost:0","resolve,listen";
    permission java.net.URLPermission "http:*","*:*";
    permission java.net.URLPermission "https:*","*:*";
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThreadGroup";
};

garyelephant avatar Jul 25 '19 06:07 garyelephant

我给es增加了一个security manager的开关。。。自己重新打包了解决的。 汗呐

wcg2008zl avatar Aug 02 '19 07:08 wcg2008zl

我给es增加了一个security manager的开关。。。自己重新打包了解决的。 汗呐

security manager开关？是运行时可动态修改吗？

请问具体怎样弄的？

microhuang avatar Aug 06 '19 03:08 microhuang

我给es增加了一个security manager的开关。。。自己重新打包了解决的。 汗呐

security manager开关？是运行时可动态修改吗？

请问具体怎样弄的？

1.es 是在Security.java里的configure方法里是设置SecurityManager的。可在在这里加自己的开关，不设置代码的操作就不会被安全策略管控 2.es会额外检查是否配置了AllPermission,BootstrapChecks.java里的内部类AllPermissionCheck来做这件事儿，也需要增加开关 3，开关运行时可修改，可以自已额外实现。无非是设置一次SecurityManager

wcg2008zl avatar Aug 07 '19 01:08 wcg2008zl

1.es 是在Security.java里的configure方法里是设置SecurityManager的。可在在这里加自己的开关，不设置代码的操作就不会被安全策略管控

按你的思路试过了，不成功。Security.configure本来就是bootstrap过程执行的，而它只在elasticsearch启动过程中被执行一次，就算在这里调整了security manager开关值，也不会对后续流程执行。

不知道是不是我的实现方式不对，方便把你的实现共享出来吗？

microhuang avatar Aug 08 '19 07:08 microhuang

  • 编写一个打印所有的权限的SecurityManager
  • 用arthas attach上去，然后就可以知道所有需要的权限了

参考代码：

https://gist.github.com/hengyunabc/c882e8609abd53836450423967a6eb27

hengyunabc avatar Aug 14 '19 07:08 hengyunabc

ES 6.2.4 不保证全部功能OK，我只测试了trace, watch, monitor这几个命令，后面应该还需要加其他permission 有两个Permission比较难找，其他的话基本可以在应用本身的stderr或者${user.home}/logs/arthas/arthas.log里面找到

  1. java.lang.reflect.ReflectPermission 当前版本的arthas(3.1.1)在启动ArthasBootstrap时，会绑定配置的telnet和http端口（默认3658和8563），这两个配置是由AgentBootstrap#bind获取到javaagent参数args转换成Configure对象来获取的。但是由于没有配置权限ReflectPermission，在调用Configure#toConfigure方法时有通过反射去设置Configure对象中对应的field，设置会失败，异常会被抓住，导致返回的Configure对象为空。详见PR #818

  2. ognl.OgnlInvokePermission 这个权限是在使用watch命令观察参数时发现的，具体错误会在${user.home}/logs/arthas/arthas.log里面

Caused by: java.lang.IllegalAccessException: Method [public java.lang.Object[] com.taobao.arthas.core.advisor.Advice.getParams()] cannot be accessed.
    at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:895) ~[arthas-core.jar:3.1.1]
    at ognl.OgnlRuntime.getMethodValue(OgnlRuntime.java:1709) ~[arthas-core.jar:3.1.1]
    at ognl.ObjectPropertyAccessor.getPossibleProperty(ObjectPropertyAccessor.java:60) ~[arthas-core.jar:3.1.1]

这边看下ognl里面的代码OgnlRuntime.java895行左右

 if (checkPermission)
{
    try
    {
        _securityManager.checkPermission(getPermission(method));
    } catch (SecurityException ex) {
        throw new IllegalAccessException("Method [" + method + "] cannot be accessed.");
    }
}

看下getPermission方法

public static Permission getPermission(Method method)
{
    Permission result;
    Class mc = method.getDeclaringClass();

    synchronized (_invokePermissionCache) {
        Map permissions = (Map) _invokePermissionCache.get(mc);

        if (permissions == null) {
            _invokePermissionCache.put(mc, permissions = new HashMap(101));
        }
        if ((result = (Permission) permissions.get(method.getName())) == null) {
            result = new OgnlInvokePermission("invoke." + mc.getName() + "." + method.getName());
            permissions.put(method.getName(), result);
        }
    }
    return result;
}

这边返回的是一个OgnlInvokePermission对象，在java.policy里面加上就OK

grant {
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThreadGroup";
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThread";
};


grant {
    permission java.io.FilePermission "<<ALL FILES>>", "read,write";
    permission java.util.PropertyPermission "JM.LOG.PATH", "write";
    permission java.lang.RuntimePermission "*";
    permission java.lang.reflect.ReflectPermission "*";
    permission java.net.SocketPermission "*", "connect,listen,resolve,accept";
    permission ognl.OgnlInvokePermission "*";
};

hydrogen666 avatar Aug 16 '19 07:08 hydrogen666

https://github.com/alibaba/arthas/pull/818

hengyunabc avatar Aug 16 '19 07:08 hengyunabc

我在es 5.4.3下面是可以的 java.policy 内容如下：

/*
 * Licensed to Elasticsearch under one or more contributor
 * license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright
 * ownership. Elasticsearch licenses this file to you under
 * the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

grant {

    permission java.security.AllPermission;
    permission java.lang.RuntimePermission "*";
    permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
    permission java.net.NetPermission "*";
    permission java.net.SocketPermission "localhost:0","resolve,listen";
    permission java.net.URLPermission "http:*","*:*";
    permission java.net.URLPermission "https:*","*:*";
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThreadGroup";
};

V5.4.3是可以的，因为5可以开启这个权限> permission java.security.AllPermission; V6.x，禁用了

Hailei avatar Apr 14 '20 06:04 Hailei

ES 6.2.4 不保证全部功能OK，我只测试了trace, watch, monitor这几个命令，后面应该还需要加其他permission 有两个Permission比较难找，其他的话基本可以在应用本身的stderr或者${user.home}/logs/arthas/arthas.log里面找到

  1. java.lang.reflect.ReflectPermission 当前版本的arthas(3.1.1)在启动ArthasBootstrap时，会绑定配置的telnet和http端口（默认3658和8563），这两个配置是由AgentBootstrap#bind获取到javaagent参数args转换成Configure对象来获取的。但是由于没有配置权限ReflectPermission，在调用Configure#toConfigure方法时有通过反射去设置Configure对象中对应的field，设置会失败，异常会被抓住，导致返回的Configure对象为空。详见PR #818
  2. ognl.OgnlInvokePermission 这个权限是在使用watch命令观察参数时发现的，具体错误会在${user.home}/logs/arthas/arthas.log里面
Caused by: java.lang.IllegalAccessException: Method [public java.lang.Object[] com.taobao.arthas.core.advisor.Advice.getParams()] cannot be accessed.
    at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:895) ~[arthas-core.jar:3.1.1]
    at ognl.OgnlRuntime.getMethodValue(OgnlRuntime.java:1709) ~[arthas-core.jar:3.1.1]
    at ognl.ObjectPropertyAccessor.getPossibleProperty(ObjectPropertyAccessor.java:60) ~[arthas-core.jar:3.1.1]

这边看下ognl里面的代码OgnlRuntime.java895行左右

 if (checkPermission)
{
    try
    {
        _securityManager.checkPermission(getPermission(method));
    } catch (SecurityException ex) {
        throw new IllegalAccessException("Method [" + method + "] cannot be accessed.");
    }
}

看下getPermission方法

public static Permission getPermission(Method method)
{
    Permission result;
    Class mc = method.getDeclaringClass();

    synchronized (_invokePermissionCache) {
        Map permissions = (Map) _invokePermissionCache.get(mc);

        if (permissions == null) {
            _invokePermissionCache.put(mc, permissions = new HashMap(101));
        }
        if ((result = (Permission) permissions.get(method.getName())) == null) {
            result = new OgnlInvokePermission("invoke." + mc.getName() + "." + method.getName());
            permissions.put(method.getName(), result);
        }
    }
    return result;
}

这边返回的是一个OgnlInvokePermission对象，在java.policy里面加上就OK

grant {
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThreadGroup";
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThread";
};


grant {
    permission java.io.FilePermission "<<ALL FILES>>", "read,write";
    permission java.util.PropertyPermission "JM.LOG.PATH", "write";
    permission java.lang.RuntimePermission "*";
    permission java.lang.reflect.ReflectPermission "*";
    permission java.net.SocketPermission "*", "connect,listen,resolve,accept";
    permission ognl.OgnlInvokePermission "*";
};

我在V6.8.0上试验过了，可用

Hailei avatar Apr 14 '20 14:04 Hailei

综合了上面几种答案 ，在 java.policy 中加上,然后就好使了。

es 5.6 arthas 3.6.1

然后需要应用启动的时候添加-D参数-Djava.security.policy=file:///path/to/java.policy

grant codeBase "file:${user.home}/.arthas/lib/-" {
    permission java.io.FilePermission "${user.home}/.arthas/-", "read,write";
    permission java.io.FilePermission "${user.home}/logs/arthas/-", "read,write,delete";
    permission java.lang.RuntimePermission "createClassLoader";
    permission java.lang.RuntimePermission "getClassLoader";
    permission java.lang.RuntimePermission "modifyThreadGroup";
    permission java.lang.RuntimePermission "modifyThread";
    permission java.lang.RuntimePermission "shutdownHooks";
    permission java.lang.RuntimePermission "accessClassInPackage.sun.reflect";
    permission java.lang.RuntimePermission "accessClassInPackage.sun.net.www.protocol.http";
    permission java.lang.RuntimePermission "accessClassInPackage.sun.net.www.http";
    permission java.net.SocketPermission "127.0.0.1:3658", "listen,resolve";
    permission java.net.SocketPermission "127.0.0.1:8563", "listen,resolve";
    permission java.net.SocketPermission "*", "accept";
};

grant {
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThreadGroup";
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThread";
};


grant {
    permission java.io.FilePermission "<<ALL FILES>>", "read,write";
    permission java.util.PropertyPermission "JM.LOG.PATH", "write";
    permission java.lang.RuntimePermission "*";
    permission java.lang.reflect.ReflectPermission "*";
    permission java.net.SocketPermission "*", "connect,listen,resolve,accept";
    permission ognl.OgnlInvokePermission "*";
};

grant {

    permission java.security.AllPermission;
    permission java.lang.RuntimePermission "*";
    permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
    permission java.net.NetPermission "*";
    permission java.net.SocketPermission "localhost:0","resolve,listen";
    permission java.net.URLPermission "http:*","*:*";
    permission java.net.URLPermission "https:*","*:*";
    permission org.elasticsearch.ThreadPermission "modifyArbitraryThreadGroup";
};

yangyu6 avatar May 25 '22 06:05 yangyu6