Randomly receive S3 error from CubeStore when building pre-aggregation data

[bongnv]

Describe the bug

When the refresh worker builds pre-aggregation data from source Athena and store them to CubeStore. We randomly receive the below error:

statusCode: 400, type: , msg: Error: Error during create table: CREATE TABLE prod_pre_aggregations.inspection_rollup_main20210601_d0of0vji_lbon14ip_1h6m94u (`inspection__author_id` varchar(255), `inspection__organisation_id` varchar(255), `inspection__owner_id` varchar(255), `inspection__site_id` varchar(255), `inspection__status` int, `inspection__template_id` varchar(255), `inspection__conducted_at_day` timestamp, `inspection__count` bigint, `inspection__count_duration` bigint, `inspection__count_score` bigint, `inspection__duration_sum` real, `inspection__score_percentage_sum` real)  WITH (input_format = 'csv_no_header') LOCATION ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?: User: Create table failed: Internal: AWS S3 error: custom: missing field `Bucket`

To Reproduce

1. Use Athena as the data source
2. Enable pre-aggregation & scheduled refresh workers
3. Receive AWS S3 error from CubeStore randomly

Expected behavior There should be no error from CubeStore.

Screenshots If applicable, add screenshots to help explain your problem.

Minimally reproducible Cube Schema N/A

Version: [0.29.28]

Additional context Add any other context about the problem here.

[vincent-benbria]

I am also experiencing this oddball behaviour randomly:

2022-09-04 07:02:31,890 ERROR [cubestore::http] <pid:1> Error processing HTTP command: User: Create table failed: User: S3 download returned non OK status: 500

2022-09-04 07:02:32,454 ERROR [cubestore::http] <pid:1> Error processing HTTP command: User: Create table failed: User: S3 download returned non OK status: 500

2022-09-05 00:05:40,675 ERROR [cubestore::util] <pid:1> Error during Meta Store Upload: CubeError { message: "AWS S3 error: custom: missing field `Bucket`", backtrace: "", cause: Internal }
2022-09-07 01:45:12,293 ERROR [cubestore::http] <pid:1> Websocket error: Protocol(ResetWithoutClosingHandshake)
2022-09-07 01:45:12,449 ERROR [cubestore::http] <pid:1> Websocket error: Protocol(ResetWithoutClosingHandshake)

Version: v0.30.60

[paveltiunov]

This error basically means that S3 fails to perform operation due to an internal server error. We should provide a better error message for that. Cube will do a retry for building this pre-aggregation in this case. Cube Store in Cube Cloud has an additional retry loop for such cases.

[tinolyu]

@paveltiunov Hi, I got the same error. What should be the better error message? For me, refresh worker was up and running, and it uploaded some files to s3. After a few hours, it started to output error of Create table failed: Internal: AWS S3 error: custom: missing field 'Bucket'. Here is my issue: https://github.com/cube-js/cube/issues/6997

[paveltiunov]

@tinolyu This error means S3 responds with a 500-503 status code, so it's inaccessible at that time.