KubeCube icon indicating copy to clipboard operation
KubeCube copied to clipboard
verified publisher icon kubecube-io

[Bug]按照文档一路安装，从1.2-到1.4【logseer】 真用不起

Open snpyeso opened this issue 3 years ago • 2 comments

继续观光一下 【Hotplug】hotplugs.hotplug.kubecube.io v1 common 与 pivot-cluster 都打开

spec:
  component:
    -
      name: audit
      status: enabled
    -
      env: "address: elasticsearch-master-headless.elasticsearch.svc\n"
      name: logseer
      namespace: logseer
      pkgName: logseer-v1.0.0.tgz
      status: enabled
    -
      env: "clustername: \"{{.cluster}}\"\n"
      name: logagent
      namespace: logagent
      pkgName: logagent-v1.0.0.tgz
      status: enabled
    -
      name: elasticsearch
      namespace: elasticsearch
      pkgName: elasticsearch-7.8.1.tgz
      status: enabled
    -
      env: "grafana:\n  enabled: false\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://172.31.0.171:31291/api/v1/receive\n"
      name: kubecube-monitoring
      namespace: kubecube-monitoring
      pkgName: kubecube-monitoring-15.4.12.tgz
      status: enabled
    -
      name: kubecube-thanos
      namespace: kubecube-monitoring
      pkgName: thanos-3.18.0.tgz
      status: enabled

spec:
  component:
    -
      env: "address: elasticsearch-master.elasticsearch.svc \n"
      name: logseer
      status: enabled
    -
      env: "grafana:\n  enabled: true\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://kubecube-thanos-receive:19291/api/v1/receive\n"
      name: kubecube-monitoring
    -
      env: "receive:\n  tsdbRetention: 7d\n  replicaCount: 1\n  replicationFactor: 1\n"
      name: kubecube-thanos
      status: enabled

默认配置，配置 elasticsearch-master-headless.elasticsearch.svc和elasticsearch-master.elasticsearch.svc 都配置过，理论上不会有什么影响，还是不行,然后进行调试

问题一：查询日志报错 “request elasticsearch fail”
问题二：操作审计无数据（经过调试已解决） 过程如下： 查看logseer运行pod的容器日志发现如下

2022-09-24 20:40:47.299 [http-nio-8080-exec-10]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:52   INFO  - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
    "size": 50,
    "from": 0,
    "query": {
      "bool" : {
        "filter" : [
            {"term": {"cluster_name" : "pivot-cluster"}},
            {"term": {"namespace" : "wordpress"}}
        ],
        "must" : [
          {
            "query_string" : {
              "default_field" : "message",
              "query" : "elasticsearch-master.elasticsearch.svc:9200"
            }
          },
          {
            "range" : {
              "@timestamp" : {
                "gte" : 1664019350313,
                "lte" : 1664022950313,
                "format": "epoch_millis"
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "2": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1m",
          "time_zone": "Asia/Shanghai",
          "min_doc_count": 1
        }
      }
    },
    "highlight" : {
      "fields" : {
        "message" : {}
      },
      "fragment_size": 2147483647
    },
    "sort" : [
      { "@timestamp" : "asc"}
    ],
    "_source" : {
      "excludes": "tags"
    },
    "timeout": "30000ms"
} 
2022-09-24 20:40:48.302 [http-nio-8080-exec-10]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:65   ERROR - request elasticsearch exception: {} 
java.net.ConnectException: null
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
	at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
	at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
	at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)

出现一个空指针异常

  • 要么取出ES的数据为空
  • 要么查找拼接的地址为空毕竟有一个没看到访问HOST地址的请求 request to es, url: /*/_search?ignore_unavailable=true 进入logseer容器

直接curl http://elasticsearch-master.elasticsearch.svc:9200//_search?ignore_unavailable=true 发现返回了一大堆数据，证明ES连通性是好的，不过毕竟没添加参数，curl不是很好加参数，加了参数可能就返回空报错了。 猜测是不是环境变量没有设置起，不停的调整环境变量格式，甚至configMap,内部配置文件 期望出现日志 equest to es, url: http://elasticsearch-master.elasticsearch.svc:9200//_search?ignore_unavailable=true。 是不是没有读取到address: elasticsearch-master.elasticsearch.svc 这个变量，最终放弃也许日志本来就是这么写的。
转入logagent 的filebeat 的configMap 发现 output.elasticsearch: hosts: [elasticsearch-master.elasticsearch.svc:30435] 这个根本访问不到修改成 output.elasticsearch: hosts: [elasticsearch-master.elasticsearch.svc:9200] 再试试，嗯一样的不通(好在的是filebeate不爆连接错误了 ) 接着看了一下文档也没发现哪里不对，再修复下审计 我本来也安装的内部ES，还是当外部配置下吧

kubectl edit deploy audit -n kubecube-system
env:
- name: AUDIT_WEBHOOK_HOST
  value: http://elasticsearch-master.elasticsearch:9200
- name: AUDIT_WEBHOOK_INDEX
  value: audit
- name: AUDIT_WEBHOOK_TYPE
  value: logs

审计可以了，

但是日志依然不通，看来只有想办法开放ES 9200端口出来用工具连连是没上传还是没查询到， 不过大体定位到如下可能的几个问题

  • 可能是logseer没有读取到环境变量导致出错
  • filebeate配置错误没有上传成功，导致查询空数据（这个数据内容空就报错应该不至于），但是确实直接安装的filebeat有一个connect 错误需要修复 费解

目前发现的问题猜测ripple和filebeat的配置感觉这里嫌疑最大，创建了新的日志抓取任务，也没看到/etc/filebeat/inputs.d 有什么文件改动 不过也建议修复下空报错的问题，让指示得更明显，只能去看看哪里有源码了

snpyeso avatar Sep 24 '22 08:09 snpyeso

应该是logseer读取不到es地址或者es地址无法访问导致的。 logseer里的配置默认应该是一个svc域名，麻烦确认一下：

  1. 域名是否正确（和es的svc对应）
  2. 域名是否被logseer读取
  3. 域名是否可解析
  4. 域名网络是否可达

ethfoo avatar Oct 24 '22 09:10 ethfoo

进入logseer容器 直接curl http://elasticsearch-master.elasticsearch.svc:9200/*/_search?ignore_unavailable=true，可有返回数据，是不是证明域名，连通性这些都是OK的

snpyeso avatar Oct 27 '22 07:10 snpyeso