ERROR [cubestore::remotefs::cleanup] error while getting created time for file - creation time is not available for the filesystem

[seatunneltrailuser]

Describe the bug While the Refresh Worker is performing pre-aggregations, we get multiple of this error

ERROR [cubestore::remotefs::cleanup] pid:1 error while getting created time for file "919-sgnl1jjo.parquet":creation time is not available for the filesystem For every intermediary file created by the pre-aggregation process, CUBESTORE is also performing cleanup of these files. It is failing to purge these files due to this issue - Creation time is not available for the filesystem.

To Reproduce Steps to reproduce the behavior:

1. Run Docker containers as following Start the container for Router in VM1 Start the container for Worker1 in VM2 Start the container for Worker2 in VM3 Start the container for RefreshWorker in VM2 Start the container for API in VM3
2. The Router, Worker1, Worker2 is configured with the CUBESTORE_REMOTE_DIR to /cube/remote, and a shared network monted volume /<network_shared_mounted_volume>/data as /cube/remote for these containers
3. Pre-aggregations are automatically started, and after sometime we can start to see the the temporary files were not cleaned up due to this error.

Expected behavior These files are successfully cleaned up by the [cubestore::remotefs::cleanup] process.

Version: Docker image cube/cubestore:latest

Additional context Is it safe to clear these files manually ? Also, Are we missing any additional parameters to be set while starting up the RefreshWorker the job ?

[igorlukanin]

error while getting created time for file "919-sgnl1jjo.parquet":creation time is not available for the filesystem

@seatunneltrailuser What are you using a the filesystem? Could you please describe your setup?

[pasqualtroncone]

I have seen the same error on my tests. My host file system is xfs.

Here is my docker compose test setup

name: "cubejs"

x-cubejs-image: &cubejs-image "docker.io/cubejs/cube:v0.35.22"
x-cubestore-image: &cubestore-image "docker.io/cubejs/cubestore:v0.35.22"

services:
  cubestore:
    image: *cubestore-image
    container_name: "cubejs-cubestore"
    hostname: "cubestore"
    networks:
      - "cubejs"
    environment:
      CUBESTORE_DATA_DIR: "/cube/data"
      CUBESTORE_LOG_LEVEL: "info"
    volumes:
      - "cubestore-data:/cube/data/"
 
  cube:
    image: *cubejs-image
    container_name: "cubejs"
    hostname: "cube"
    networks:
      - "cubejs"
    ports:
      - "4000:4000/tcp"
      - "5000:5000/tcp"
    environment:
      CUBEJS_CUBESTORE_HOST: "cubestore"
      CUBEJS_DB_HOST: "${CUBEJS_DB_HOST}"
      CUBEJS_DB_PORT: "${CUBEJS_DB_PORT}"
      CUBEJS_DB_NAME: "${CUBEJS_DB_NAME}"
      CUBEJS_DB_USER: "${CUBEJS_DB_USER}"
      CUBEJS_DB_PASS: "${CUBEJS_DB_PASS}"
      CUBEJS_DB_TYPE: "${CUBEJS_DB_TYPE}"
      CUBEJS_DB_CLICKHOUSE_READONLY: "true"
      CUBEJS_API_SECRET: "${CUBEJS_API_SECRET}"
      CUBEJS_PG_SQL_PORT: "${CUBEJS_PG_SQL_PORT}"
      CUBEJS_SQL_USER: "${CUBEJS_SQL_USER}"
      CUBEJS_SQL_PASSWORD: "${CUBEJS_SQL_PASSWORD}"
      CUBEJS_CACHE_AND_QUEUE_DRIVER: "cubestore"
      CUBEJS_LOG_LEVEL: "info"
      CUBEJS_DEV_MODE: "true"
    volumes:
      - "cube-data:/cube/conf/"
      - "./config/cubejs/cube.js:/cube/conf/cube.js:ro"
      - "./config/cubejs/model/:/cube/conf/model/custom/"
    depends_on:
      - "cubestore"

volumes:
  cube-data:
    name: "cubejs-cube-data"

  cubestore-data:
    name: "cubejs-cubestore-data"

networks:

  cubejs:
    name: "cubejs"
    internal: false

Log extracts

cubejs-cubestore  | 2024-04-24T00:08:44.467Z INFO  [cubestore::metastore::rocks_store] <pid:1> Persisting cachestore snapshot: done (2.25434ms)
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "1-hnwpuxtq.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "2-3nyem971.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "3-lujmbedj.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "4-xoct54kf.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "5-t2ipzm33.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "6-kujmssqi.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "8-i2idmyai.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "7-6hnq8uxm.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "9-c0fsdovn.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "10-ge6ds2ub.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "11-lcnahpyk.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "12-u7wwba87.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "14-fihpqpdt.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "13-zv3tgnxa.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "15-ncwvy49u.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "16-rfrsdovy.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "18-npj05axs.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "17-7ybmvqyi.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "19-xvwdfesw.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "20-8a0pftmc.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "22-cp3fonx5.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "21-swbvm5no.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "23-levsfozf.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "24-xu3nzyfw.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "26-emndusua.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "25-q9klsjyf.chunk.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "28-jvt5sxb2.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "29-oqc1wfzn.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "30-zv3g3peo.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "27-yia1mvib.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "31-wqi37b0k.parquet":creation time is not available for the filesystem
cubejs-cubestore  | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] <pid:1> error while getting created time for file "32-8lhd05kd.parquet":creation time is not available for the filesystem