seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard
verified publisher icon apache

[Bug] [engine] OSS storage reports NoSuchKey

Open Asura7969 opened this issue 1 year ago • 0 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

checkpoint storage: aliyun OSS

Not sure if this log is normal，it doesn't seem to affect the task

2024-02-22 15:36:25,300 INFO  [c.a.oss                       ] [seatunnel-coordinator-service-5] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 65D6F9792316833136BB3609
[HostId]: null

SeaTunnel Version

2.3.4

SeaTunnel Config

seatunnel:
  engine:
    history-job-expire-minutes: 1440
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: true
    checkpoint:
      interval: 10000
      timeout: 60000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          storage.type: oss
          namespace: /tmp/seatunnel/checkpoint_snapshot
          oss.bucket: oss://xxxx/              
          fs.oss.accessKeyId: xxxx
          fs.oss.accessKeySecret: xxxx
          fs.oss.endpoint: xxxx
          fs.oss.credentials.provider: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider

Running Command

mysql cdc to doris

Error Exception

2024-02-22 15:36:25,300 INFO  [c.a.oss                       ] [seatunnel-coordinator-service-5] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 65D6F9792316833136BB3609
[HostId]: null
2024-02-22 15:36:25,301 DEBUG [o.a.h.c.p.RequestAddCookies   ] [seatunnel-coordinator-service-5] - CookieSpec selected: default
2024-02-22 15:36:25,301 DEBUG [o.a.h.c.p.RequestAuthCache    ] [seatunnel-coordinator-service-5] - Auth cache not set in the context
2024-02-22 15:36:25,301 DEBUG [ingHttpClientConnectionManager] [seatunnel-coordinator-service-5] - Connection request: [route: {s}->https://ossname.oss-cn-hangzhou.aliyuncs.com:443][total kept alive: 1; route allocated: 1 of 32; total allocated: 1 of 32]
2024-02-22 15:36:25,301 DEBUG [ingHttpClientConnectionManager] [seatunnel-coordinator-service-5] - Connection leased: [id: 8][route: {s}->https://ossname.oss-cn-hangzhou.aliyuncs.com:443][total kept alive: 0; route allocated: 1 of 32; total allocated: 1 of 32]
2024-02-22 15:36:25,301 DEBUG [ultManagedHttpClientConnection] [seatunnel-coordinator-service-5] - http-outgoing-8: set socket timeout to 200000
2024-02-22 15:36:25,301 DEBUG [o.a.h.i.e.MainClientExec      ] [seatunnel-coordinator-service-5] - Executing request HEAD /tmp/seatunnel/checkpoint_snapshot/812949238777774092/1708587384896-135-1-29.sertmp/ HTTP/1.1
2024-02-22 15:36:25,301 DEBUG [o.a.h.i.e.MainClientExec      ] [seatunnel-coordinator-service-5] - Proxy auth state: UNCHALLENGED
2024-02-22 15:36:25,301 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> HEAD /tmp/seatunnel/checkpoint_snapshot/812949238777774092/1708587384896-135-1-29.sertmp/ HTTP/1.1
2024-02-22 15:36:25,301 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> Authorization: OSS {oss.accessKeyId}:VQff7GpQCjWx9mC21p1brjTE74A=
2024-02-22 15:36:25,301 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> Date: Thu, 22 Feb 2024 07:36:25 GMT
2024-02-22 15:36:25,301 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> Host: ossname.oss-cn-hangzhou.aliyuncs.com
2024-02-22 15:36:25,301 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> Connection: Keep-Alive
2024-02-22 15:36:25,301 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> User-Agent: aliyun-sdk-java/3.4.1(Linux/3.10.0-1160.105.1.el7.x86_64/amd64;1.8.0_401), Hadoop/3.1.4
2024-02-22 15:36:25,301 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> "HEAD /tmp/seatunnel/checkpoint_snapshot/812949238777774092/1708587384896-135-1-29.sertmp/ HTTP/1.1[\r][\n]"
2024-02-22 15:36:25,301 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> "Authorization: OSS {oss.accessKeyId}:VQff7GpQCjWx9mC21p1brjTE74A=[\r][\n]"
2024-02-22 15:36:25,301 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> "Date: Thu, 22 Feb 2024 07:36:25 GMT[\r][\n]"
2024-02-22 15:36:25,301 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> "Host: ossname.oss-cn-hangzhou.aliyuncs.com[\r][\n]"
2024-02-22 15:36:25,301 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> "Connection: Keep-Alive[\r][\n]"
2024-02-22 15:36:25,301 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> "User-Agent: aliyun-sdk-java/3.4.1(Linux/3.10.0-1160.105.1.el7.x86_64/amd64;1.8.0_401), Hadoop/3.1.4[\r][\n]"
2024-02-22 15:36:25,301 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 >> "[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "HTTP/1.1 404 Not Found[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "Server: AliyunOSS[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "Date: Thu, 22 Feb 2024 07:36:25 GMT[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "Content-Type: application/xml[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "Content-Length: 455[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "Connection: keep-alive[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "x-oss-request-id: 65D6F9792316833136343709[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "Vary: Origin[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "x-oss-server-time: 2[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "x-oss-ec: 0026-00000001[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "x-oss-err: PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPEVycm9yPgogIDxDb2RlPk5vU3VjaEtleTwvQ29kZT4KICA8TWVzc2FnZT5UaGUgc3BlY2lmaWVkIGtleSBkb2VzIG5vdCBleGlzdC48L01lc3NhZ2U+CiAgPFJlcXVlc3RJZD42NUQ2Rjk3OTIzMTY4MzMxMzYzNDM3MDk8L1JlcXVlc3RJZD4KICA8SG9zdElkPnNjLXN0cmF0ZWd5LW1hcC5vc3MtY24taGFuZ3pob3UuYWxpeXVuY3MuY29tPC9Ib3N0SWQ+CiAgPEtleT50bXAvc2VhdHVubmVsL2NoZWNrcG9pbnRfc25hcHNob3QvODEyOTQ5MjM4Nzc3Nzc0MDkyLzE3MDg1ODczODQ4OTYtMTM1LTEtMjkuc2VydG1wLzwvS2V5PgogIDxFQz4wMDI2LTAwMDAwMDAxPC9FQz4KICA8UmVjb21tZW5kRG9jPmh0dHBzOi8vYXBpLmFsaXl1bi5jb20vdHJvdWJsZXNob290P3E9MDAyNi0wMDAwMDAwMTwvUmVjb21tZW5kRG9jPgo8L0Vycm9yPgo=[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.wire                    ] [seatunnel-coordinator-service-5] - http-outgoing-8 << "[\r][\n]"
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << HTTP/1.1 404 Not Found
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << Server: AliyunOSS
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << Date: Thu, 22 Feb 2024 07:36:25 GMT
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << Content-Type: application/xml
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << Content-Length: 455
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << Connection: keep-alive
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << x-oss-request-id: 65D6F9792316833136343709
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << Vary: Origin
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << x-oss-server-time: 2
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << x-oss-ec: 0026-00000001
2024-02-22 15:36:25,318 DEBUG [o.a.h.headers                 ] [seatunnel-coordinator-service-5] - http-outgoing-8 << x-oss-err: PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPEVycm9yPgogIDxDb2RlPk5vU3VjaEtleTwvQ29kZT4KICA8TWVzc2FnZT5UaGUgc3BlY2lmaWVkIGtleSBkb2VzIG5vdCBleGlzdC48L01lc3NhZ2U+CiAgPFJlcXVlc3RJZD42NUQ2Rjk3OTIzMTY4MzMxMzYzNDM3MDk8L1JlcXVlc3RJZD4KICA8SG9zdElkPnNjLXN0cmF0ZWd5LW1hcC5vc3MtY24taGFuZ3pob3UuYWxpeXVuY3MuY29tPC9Ib3N0SWQ+CiAgPEtleT50bXAvc2VhdHVubmVsL2NoZWNrcG9pbnRfc25hcHNob3QvODEyOTQ5MjM4Nzc3Nzc0MDkyLzE3MDg1ODczODQ4OTYtMTM1LTEtMjkuc2VydG1wLzwvS2V5PgogIDxFQz4wMDI2LTAwMDAwMDAxPC9FQz4KICA8UmVjb21tZW5kRG9jPmh0dHBzOi8vYXBpLmFsaXl1bi5jb20vdHJvdWJsZXNob290P3E9MDAyNi0wMDAwMDAwMTwvUmVjb21tZW5kRG9jPgo8L0Vycm9yPgo=
2024-02-22 15:36:25,318 DEBUG [o.a.h.i.e.MainClientExec      ] [seatunnel-coordinator-service-5] - Connection can be kept alive indefinitely
2024-02-22 15:36:25,318 DEBUG [ingHttpClientConnectionManager] [seatunnel-coordinator-service-5] - Connection [id: 8][route: {s}->https://ossname.oss-cn-hangzhou.aliyuncs.com:443] can be kept alive indefinitely
2024-02-22 15:36:25,318 DEBUG [ingHttpClientConnectionManager] [seatunnel-coordinator-service-5] - Connection released: [id: 8][route: {s}->https://ossname.oss-cn-hangzhou.aliyuncs.com:443][total kept alive: 1; route allocated: 1 of 32; total allocated: 1 of 32]
2024-02-22 15:36:25,318 INFO  [c.a.oss                       ] [seatunnel-coordinator-service-5] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 65D6F9792316833136343709
[HostId]: null

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

Asura7969 avatar Feb 22 '24 07:02 Asura7969