apisix icon indicating copy to clipboard operation
apisix copied to clipboard

Published 20 hours ago • verified publisher icon apache

help request: log_rotate is not working

Open AmerDwight opened this issue 1 year ago • 4 comments

Description

I'm trying to active log_rotate, but is not working. After reviewing the error.log , I guess log_rotate is not load at first time. Is there anything I missed?

Apisix config.yaml:

apisix:
  node_listen: 9080              # APISIX listening port
  enable_ipv6: false

  enable_control: true
  control:
    ip: "0.0.0.0"
    port: 9092

deployment:
  admin:
    allow_admin:               # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
      - 0.0.0.0/0              # We need to restrict ip access rules for security. 0.0.0.0/0 is for test.

    admin_listen:
      port: 9180

    admin_key:
      - name: "admin"
        key: edd1c9f034335f136f87ad84b625c8f1
        role: admin                 # admin: manage all configuration data

      - name: "viewer"
        key: 4054f7cf07e344346cd3f287985e76a2
        role: viewer

  etcd:
    host:                           # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
      - "http://etcd:2379"          # multiple etcd address
    prefix: "/apisix"               # apisix configurations prefix
    timeout: 30                     # 30 seconds

  plugins:
    - log-rotate
  plugin_attr:
    file-logger:
      path: "./logs/file.logger/daily.log"
    log_rotate:
      interval: 60
      max_kept: 168
      max_size: 36000
      # either restrict log by time or size, still not work
      
    prometheus:
      metrics:
          http_status:
              extra_labels:
                  - upstream_addr: $upstream_addr
                  - upstream_status: $upstream_status
      export_addr:
        ip: "0.0.0.0"
        port: 9091

error.log: there is no priority 100 name: log_rotate

2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:202: load(): new plugins: {"basic-auth":true,"openfunction":true,"client-control":true,"serverless-pre-function":true,"ai":true,"rocketmq-logger":true,"grpc-transcode":true,"google-cloud-logging":true,"api-breaker":true,"limit-req":true,"splunk-hec-logging":true,"public-api":true,"udp-logger":true,"degraphql":true,"server-info":true,"loggly":true,"cors":true,"inspect":true,"chaitin-waf":true,"loki-logger":true,"redirect":true,"referer-restriction":true,"grpc-web":true,"ext-plugin-pre-req":true,"body-transformer":true,"jwt-auth":true,"tcp-logger":true,"proxy-rewrite":true,"proxy-control":true,"hmac-auth":true,"elasticsearch-logger":true,"kafka-logger":true,"http-logger":true,"real-ip":true,"serverless-post-function":true,"ldap-auth":true,"syslog":true,"skywalking-logger":true,"ext-plugin-post-resp":true,"limit-conn":true,"limit-count":true,"openwhisk":true,"mocking":true,"cas-auth":true,"authz-casdoor":true,"azure-functions":true,"aws-lambda":true,"traffic-split":true,"echo":true,"request-id":true,"datadog":true,"uri-blocker":true,"clickhouse-logger":true,"file-logger":true,"proxy-mirror":true,"consumer-restriction":true,"authz-keycloak":true,"zipkin":true,"authz-casbin":true,"opa":true,"key-auth":true,"csrf":true,"response-rewrite":true,"proxy-cache":true,"fault-injection":true,"workflow":true,"example-plugin":true,"ua-restriction":true,"ip-restriction":true,"forward-auth":true,"prometheus":true,"request-validation":true,"ext-plugin-post-req":true,"gzip":true,"kafka-proxy":true,"tencent-cloud-cls":true,"wolf-rbac":true,"openid-connect":true,"sls-logger":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 54#54: *5 [lua] plugin.lua:202: load(): new plugins: {"basic-auth":true,"openfunction":true,"client-control":true,"serverless-pre-function":true,"ai":true,"rocketmq-logger":true,"grpc-transcode":true,"google-cloud-logging":true,"api-breaker":true,"limit-req":true,"splunk-hec-logging":true,"public-api":true,"udp-logger":true,"degraphql":true,"server-info":true,"loggly":true,"cors":true,"inspect":true,"chaitin-waf":true,"loki-logger":true,"redirect":true,"referer-restriction":true,"grpc-web":true,"ext-plugin-pre-req":true,"body-transformer":true,"jwt-auth":true,"tcp-logger":true,"proxy-rewrite":true,"proxy-control":true,"hmac-auth":true,"elasticsearch-logger":true,"kafka-logger":true,"http-logger":true,"real-ip":true,"serverless-post-function":true,"ldap-auth":true,"syslog":true,"skywalking-logger":true,"ext-plugin-post-resp":true,"limit-conn":true,"limit-count":true,"openwhisk":true,"mocking":true,"cas-auth":true,"authz-casdoor":true,"azure-functions":true,"aws-lambda":true,"traffic-split":true,"echo":true,"request-id":true,"datadog":true,"uri-blocker":true,"clickhouse-logger":true,"file-logger":true,"proxy-mirror":true,"consumer-restriction":true,"authz-keycloak":true,"zipkin":true,"authz-casbin":true,"opa":true,"key-auth":true,"csrf":true,"response-rewrite":true,"proxy-cache":true,"fault-injection":true,"workflow":true,"example-plugin":true,"ua-restriction":true,"ip-restriction":true,"forward-auth":true,"prometheus":true,"request-validation":true,"ext-plugin-post-req":true,"gzip":true,"kafka-proxy":true,"tencent-cloud-cls":true,"wolf-rbac":true,"openid-connect":true,"sls-logger":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:202: load(): new plugins: {"basic-auth":true,"openfunction":true,"client-control":true,"serverless-pre-function":true,"ai":true,"rocketmq-logger":true,"grpc-transcode":true,"google-cloud-logging":true,"api-breaker":true,"limit-req":true,"splunk-hec-logging":true,"public-api":true,"udp-logger":true,"degraphql":true,"server-info":true,"loggly":true,"cors":true,"inspect":true,"chaitin-waf":true,"loki-logger":true,"redirect":true,"referer-restriction":true,"grpc-web":true,"ext-plugin-pre-req":true,"body-transformer":true,"jwt-auth":true,"tcp-logger":true,"proxy-rewrite":true,"proxy-control":true,"hmac-auth":true,"elasticsearch-logger":true,"kafka-logger":true,"http-logger":true,"real-ip":true,"serverless-post-function":true,"ldap-auth":true,"syslog":true,"skywalking-logger":true,"ext-plugin-post-resp":true,"limit-conn":true,"limit-count":true,"openwhisk":true,"mocking":true,"cas-auth":true,"authz-casdoor":true,"azure-functions":true,"aws-lambda":true,"traffic-split":true,"echo":true,"request-id":true,"datadog":true,"uri-blocker":true,"clickhouse-logger":true,"file-logger":true,"proxy-mirror":true,"consumer-restriction":true,"authz-keycloak":true,"zipkin":true,"authz-casbin":true,"opa":true,"key-auth":true,"csrf":true,"response-rewrite":true,"proxy-cache":true,"fault-injection":true,"workflow":true,"example-plugin":true,"ua-restriction":true,"ip-restriction":true,"forward-auth":true,"prometheus":true,"request-validation":true,"ext-plugin-post-req":true,"gzip":true,"kafka-proxy":true,"tencent-cloud-cls":true,"wolf-rbac":true,"openid-connect":true,"sls-logger":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:202: load(): new plugins: {"basic-auth":true,"openfunction":true,"client-control":true,"serverless-pre-function":true,"ai":true,"rocketmq-logger":true,"grpc-transcode":true,"google-cloud-logging":true,"api-breaker":true,"limit-req":true,"splunk-hec-logging":true,"public-api":true,"udp-logger":true,"degraphql":true,"server-info":true,"loggly":true,"cors":true,"inspect":true,"chaitin-waf":true,"loki-logger":true,"redirect":true,"referer-restriction":true,"grpc-web":true,"ext-plugin-pre-req":true,"body-transformer":true,"jwt-auth":true,"tcp-logger":true,"proxy-rewrite":true,"proxy-control":true,"hmac-auth":true,"elasticsearch-logger":true,"kafka-logger":true,"http-logger":true,"real-ip":true,"serverless-post-function":true,"ldap-auth":true,"syslog":true,"skywalking-logger":true,"ext-plugin-post-resp":true,"limit-conn":true,"limit-count":true,"openwhisk":true,"mocking":true,"cas-auth":true,"authz-casdoor":true,"azure-functions":true,"aws-lambda":true,"traffic-split":true,"echo":true,"request-id":true,"datadog":true,"uri-blocker":true,"clickhouse-logger":true,"file-logger":true,"proxy-mirror":true,"consumer-restriction":true,"authz-keycloak":true,"zipkin":true,"authz-casbin":true,"opa":true,"key-auth":true,"csrf":true,"response-rewrite":true,"proxy-cache":true,"fault-injection":true,"workflow":true,"example-plugin":true,"ua-restriction":true,"ip-restriction":true,"forward-auth":true,"prometheus":true,"request-validation":true,"ext-plugin-post-req":true,"gzip":true,"kafka-proxy":true,"tencent-cloud-cls":true,"wolf-rbac":true,"openid-connect":true,"sls-logger":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 54#54: *5 [lua] plugin.lua:252: load_stream(): new plugins: {"syslog":true,"ip-restriction":true,"limit-conn":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:202: load(): new plugins: {"basic-auth":true,"openfunction":true,"client-control":true,"serverless-pre-function":true,"ai":true,"rocketmq-logger":true,"grpc-transcode":true,"google-cloud-logging":true,"api-breaker":true,"limit-req":true,"splunk-hec-logging":true,"public-api":true,"udp-logger":true,"degraphql":true,"server-info":true,"loggly":true,"cors":true,"inspect":true,"chaitin-waf":true,"loki-logger":true,"redirect":true,"referer-restriction":true,"grpc-web":true,"ext-plugin-pre-req":true,"body-transformer":true,"jwt-auth":true,"tcp-logger":true,"proxy-rewrite":true,"proxy-control":true,"hmac-auth":true,"elasticsearch-logger":true,"kafka-logger":true,"http-logger":true,"real-ip":true,"serverless-post-function":true,"ldap-auth":true,"syslog":true,"skywalking-logger":true,"ext-plugin-post-resp":true,"limit-conn":true,"limit-count":true,"openwhisk":true,"mocking":true,"cas-auth":true,"authz-casdoor":true,"azure-functions":true,"aws-lambda":true,"traffic-split":true,"echo":true,"request-id":true,"datadog":true,"uri-blocker":true,"clickhouse-logger":true,"file-logger":true,"proxy-mirror":true,"consumer-restriction":true,"authz-keycloak":true,"zipkin":true,"authz-casbin":true,"opa":true,"key-auth":true,"csrf":true,"response-rewrite":true,"proxy-cache":true,"fault-injection":true,"workflow":true,"example-plugin":true,"ua-restriction":true,"ip-restriction":true,"forward-auth":true,"prometheus":true,"request-validation":true,"ext-plugin-post-req":true,"gzip":true,"kafka-proxy":true,"tencent-cloud-cls":true,"wolf-rbac":true,"openid-connect":true,"sls-logger":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 23000 name: real-ip, context: init_worker_by_lua*

<!-- Skip -->

2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 397 name: tencent-cloud-cls, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 200 name: inspect, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 0 name: example-plugin, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: -1899 name: aws-lambda, context: init_worker_by_lua*

<!-- Skip -->

2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:252: load_stream(): new plugins: {"syslog":true,"ip-restriction":true,"limit-conn":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 3000 name: ip-restriction, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 1003 name: limit-conn, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 1000 name: mqtt-proxy, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 49#49: *3 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 401 name: syslog, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 23000 name: real-ip, context: init_worker_by_lua*

<!-- Skip -->

2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 200 name: inspect, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 0 name: example-plugin, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: -1899 name: aws-lambda, context: init_worker_by_lua*

<!-- Skip -->

2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:252: load_stream(): new plugins: {"syslog":true,"ip-restriction":true,"limit-conn":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 3000 name: ip-restriction, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 1003 name: limit-conn, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 1000 name: mqtt-proxy, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 50#50: *1 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 401 name: syslog, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 54#54: *19 [lua] config_etcd.lua:171: main etcd watcher started, revision=1231, context: ngx.timer
2024/02/27 06:27:14 [warn] 49#49: *36 [lua] config_etcd.lua:171: main etcd watcher started, revision=1231, context: ngx.timer
2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 23000 name: real-ip, context: init_worker_by_lua*

<!-- Skip -->

2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 200 name: inspect, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 0 name: example-plugin, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: -1899 name: aws-lambda, context: init_worker_by_lua*

<!-- Skip -->

2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:252: load_stream(): new plugins: {"syslog":true,"ip-restriction":true,"limit-conn":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 3000 name: ip-restriction, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 1003 name: limit-conn, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 1000 name: mqtt-proxy, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 48#48: *2 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 401 name: syslog, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 50#50: *53 [lua] config_etcd.lua:171: main etcd watcher started, revision=1231, context: ngx.timer
2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 23000 name: real-ip, context: init_worker_by_lua*

<!-- Skip -->

4/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 200 name: inspect, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: 0 name: example-plugin, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:232: load(): loaded plugin and sort by priority: -1899 name: aws-lambda, context: init_worker_by_lua*

<!-- Skip -->

2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:252: load_stream(): new plugins: {"syslog":true,"ip-restriction":true,"limit-conn":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 3000 name: ip-restriction, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 1003 name: limit-conn, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 1000 name: mqtt-proxy, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 51#51: *4 [lua] plugin.lua:273: load_stream(): loaded stream plugin and sort by priority: 401 name: syslog, context: init_worker_by_lua*
2024/02/27 06:27:14 [warn] 48#48: *74 [lua] config_etcd.lua:171: main etcd watcher started, revision=1231, context: ngx.timer
2024/02/27 06:27:14 [warn] 51#51: *92 [lua] config_etcd.lua:171: main etcd watcher started, revision=1231, context: ngx.timer

this is run on docker: docker-compose:

version: "3"

services:
  apisix-dashboard:
    container_name: apisix_dashboard
    image: apache/apisix-dashboard:latest
    restart: always
    volumes:
    - ./dashboard_conf/conf.yaml:/usr/local/apisix-dashboard/conf/conf.yaml
    ports:
    - "9000:9000"
    networks:
      apisix:

  apisix:
    container_name: apisix
    image: apache/apisix:3.6.0-centos
    restart: always
    volumes:
      - ./apisix_log:/usr/local/apisix/logs
      - ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
      - ./apisix_conf/debug.yaml:/usr/local/apisix/conf/debug.yaml:ro
      - ./apisix_conf/customized/upstream.lua:/usr/local/apisix/apisix/upstream.lua:ro
      # - ./apisix_conf/customized/plugins/log-rotate.lua:/usr/local/apisix/apisix/plugins/log-rotate.lua:ro
      # - ./apisix_conf/customized/ngx_tpl.lua:/usr/local/apisix/apisix/cli/ngx_tpl.lua:ro
    depends_on:
      - etcd
    ports:
      - "9180:9180/tcp"
      - "9080:9080/tcp"
      - "9091:9091/tcp"
      - "9443:9443/tcp"
      - "9092:9092/tcp"
    networks:
      apisix:
    logging:
      options:
        max-size: "10m"
        max-file: "3"

  etcd:
    container_name: etcd
    image: bitnami/etcd:3.5
    restart: always
    volumes:
      - etcd_data:/bitnami/etcd
    environment:
      ETCD_ENABLE_V2: "true"
      ALLOW_NONE_AUTHENTICATION: "yes"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd:2379"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
    ports:
      - "12379:2379/tcp"
    networks:
      apisix:


networks:
  apisix:
    driver: bridge

volumes:
  etcd_data:
    driver: local
  alertmanager-data:
    driver: local

Environment

  • APISIX version (run apisix version): 3.6.0-centos ( official docker-image)
  • Operating system (run uname -a): docker
  • etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): etcd:3.5

AmerDwight avatar Feb 27 '24 06:02 AmerDwight

After reviewing the error.log , I guess log_rotate is not load at first time.

This is the default behaviour. Why do you say it is not working?

shreemaan-abhishek avatar Feb 29 '24 03:02 shreemaan-abhishek

Cause the rotataion of log is not actually work. Size of access.log & error.log keep growing up. And it dosen't rotate like official sample here

AmerDwight avatar Feb 29 '24 05:02 AmerDwight

#10795 see my answer！ If you want to enable a plug-in such as log-rotate, you must assign the full plug-in configuration to the custom configuration file, or enable it in config-default.yaml. Only opening a plug-in in a custom configuration will cause failure

season886 avatar Apr 20 '24 01:04 season886

@season886 So nice! I will try later. Thank you!!

AmerDwight avatar Apr 22 '24 06:04 AmerDwight

@AmerDwight any update? Is this issue still active?

kayx23 avatar Jun 13 '24 12:06 kayx23

@kayx23 The answer from @season886 is helpful. Thanks.

AmerDwight avatar Jun 18 '24 00:06 AmerDwight