[VL] random issue on GHA: terminate called without an active exception

[zhouyuan]

Backend

VL (Velox)

Bug description

VeloxOrcDataTypeValidationSuite:
E20240509 09:27:33.494535 61272 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1850, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
E20240509 09:27:33.781136 61272 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1850, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
- Bool type
E20240509 09:27:34.375708 61272 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1850, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
E20240509 09:27:34.512818 58314 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlan.cc:1852, Function:setFilterInfo, Expression:  Subfield filters creation not supported for input type 'VARBINARY' in setFilterInfo, Source: RUNTIME, ErrorCode: NOT_IMPLEMENTED
E20240509 09:27:34.544896 58314 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlan.cc:1852, Function:setFilterInfo, Expression:  Subfield filters creation not supported for input type 'VARBINARY' in setFilterInfo, Source: RUNTIME, ErrorCode: NOT_IMPLEMENTED
E20240509 09:27:34.577921 58314 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlan.cc:1852, Function:setFilterInfo, Expression:  Subfield filters creation not supported for input type 'VARBINARY' in setFilterInfo, Source: RUNTIME, ErrorCode: NOT_IMPLEMENTED
E20240509 09:27:34.607234 58314 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlan.cc:1852, Function:setFilterInfo, Expression:  Subfield filters creation not supported for input type 'VARBINARY' in setFilterInfo, Source: RUNTIME, ErrorCode: NOT_IMPLEMENTED
E20240509 09:27:34.827401 58314 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlan.cc:1852, Function:setFilterInfo, Expression:  Subfield filters creation not supported for input type 'VARBINARY' in setFilterInfo, Source: RUNTIME, ErrorCode: NOT_IMPLEMENTED
E20240509 09:27:34.860780 58314 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlan.cc:1852, Function:setFilterInfo, Expression:  Subfield filters creation not supported for input type 'VARBINARY' in setFilterInfo, Source: RUNTIME, ErrorCode: NOT_IMPLEMENTED
E20240509 09:27:34.894490 58314 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlan.cc:1852, Function:setFilterInfo, Expression:  Subfield filters creation not supported for input type 'VARBINARY' in setFilterInfo, Source: RUNTIME, ErrorCode: NOT_IMPLEMENTED
E20240509 09:27:34.930202 58314 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlan.cc:1852, Function:setFilterInfo, Expression:  Subfield filters creation not supported for input type 'VARBINARY' in setFilterInfo, Source: RUNTIME, ErrorCode: NOT_IMPLEMENTED
- Binary type
E20240509 09:27:35.139830 61272 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1850, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
- String type
E20240509 09:27:35.702018 61272 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1850, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
- Double type
E20240509 09:27:36.363720 61272 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1850, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
- Float type
E20240509 09:27:36.987668 61272 Exceptions.h:69] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1850, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
- Long type
pure virtual method called
terminate called without an active exception

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[zhztheplayer]

I observed this too as early as before https://github.com/apache/incubator-gluten/pull/5658 was merged. Will check again if the PR change is related.

[zhztheplayer]

I observed this too as early as before #5658 was merged. Will check again if the PR change is related.

I did a check and think it should not relate to #5658.

[FelixYBW]

I met the same issue, no extra info. The error log:

24/06/24 22:39:56 INFO [Executor task launch worker for task 988.0 in stage 10.0 (TID 311445)] codec.CodecConfig: Compression: ZSTD
24/06/24 22:39:56 INFO [Executor task launch worker for task 1582.0 in stage 10.0 (TID 312039)] codec.CodecConfig: Compression: ZSTD
terminate called without an active exception

It's not killed by Yarn or system due to OOM. only 10% memory is used when error happens