trino icon indicating copy to clipboard operation
trino copied to clipboard

Handle Delta Lake vacuum file system error

Open nevillelyh opened this issue 1 year ago • 5 comments

Description

Error when deleting S3 files should be catagorized as file system error for accounting purposes.

Stack trace:

{
  "type": "io.trino.spi.TrinoException",
  "message": "Failure when vacuuming <table> with retention 7d: java.io.IOException: Exception while batch deleting paths",
  "cause": {
    "type": "java.lang.RuntimeException",
    "message": "Failure when vacuuming <table> with retention 7d: java.io.IOException: Exception while batch deleting paths",
    "cause": {
      "type": "java.io.IOException",
      "message": "Exception while batch deleting paths",
      "cause": {
        "type": "com.amazonaws.services.s3.model.MultiObjectDeleteException",
        "message": "One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: <request_id>; S3 Extended Request ID: <request_id> ; Proxy: null)",
        "suppressed": [],
        "stack": [
          "com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2388)",
          "io.trino.hdfs.s3.TrinoS3FileSystem.deletePaths(TrinoS3FileSystem.java:819)",
          "io.trino.hdfs.s3.TrinoS3FileSystem.deleteFiles(TrinoS3FileSystem.java:799)",
          "io.trino.filesystem.hdfs.HdfsFileSystem.lambda$deleteFiles$2(HdfsFileSystem.java:139)",
          "io.trino.hdfs.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:25)",
          "io.trino.hdfs.HdfsEnvironment.doAs(HdfsEnvironment.java:134)",
          "io.trino.filesystem.hdfs.HdfsFileSystem.deleteFiles(HdfsFileSystem.java:137)",
          "io.trino.filesystem.manager.SwitchingFileSystem.deleteFiles(SwitchingFileSystem.java:88)",
          "io.trino.filesystem.tracing.TracingFileSystem.lambda$deleteFiles$1(TracingFileSystem.java:79)",
          "io.trino.filesystem.tracing.Tracing.lambda$withTracing$1(Tracing.java:38)",
          "io.trino.filesystem.tracing.Tracing.withTracing(Tracing.java:47)",
          "io.trino.filesystem.tracing.Tracing.withTracing(Tracing.java:37)",
          "io.trino.filesystem.tracing.TracingFileSystem.deleteFiles(TracingFileSystem.java:79)",
          "io.trino.plugin.deltalake.procedure.VacuumProcedure.doVacuum(VacuumProcedure.java:296)",
          "io.trino.plugin.deltalake.procedure.VacuumProcedure.vacuum(VacuumProcedure.java:135)",
          "io.trino.plugin.objectstore.MethodHandles$Translate.invoke(MethodHandles.java:83)",
          "java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)",
          "java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:758)",
          "io.trino.execution.CallTask.execute(CallTask.java:210)",
          "io.trino.execution.CallTask.execute(CallTask.java:70)",
...

Additional context and related issues

Release notes

(x) This is not user-visible or is docs only, and no release notes are required. ( ) Release notes are required. Please propose a release note for me. ( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

nevillelyh avatar Jun 03 '24 17:06 nevillelyh

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

github-actions[bot] avatar Jul 04 '24 17:07 github-actions[bot]

fyi @findinpath

mosabua avatar Jul 04 '24 18:07 mosabua

@ebyhr is there anything that we should do extra here? I see you did in the meantime 7a863b30c1448024feb92bbd6119943d977f80b5 to expose more information about the error.

Is storage an internal part of Trino (legalizing INTERNAL_ERROR) or rather an external part?

cc @electrum @mayankvadariya @pajaks

findinpath avatar Jul 05 '24 09:07 findinpath