Example FE CN Shared Data Seems to not work

[inviscid]

Describe the bug

I attempted to follow the example for the shared storage setup after deploying the operator. The shared storage example only deploys the FE and CN components with bucket path and credentials. Our desired approach is to use GCS to persist the data.

Things to note:

• Since we use ArgoCD for deployments, the operator is NOT installed in the starrocks namespace but rather in cluster-components namespace. It seems to start up and the starrocks clusters deploy so I hope this is not an issue.
• GCS bucket using S3 protocol - Verified the credentials allow full control of bucket
• GCS bucket is empty - we have a path specified in the config. This path does not exist yet since the bucket is empty.
• The example has a minor typo which was easily fixed enable_load_volume_from_conf = ture to true

To Reproduce

Deploy the operator on a GKE cluster using a GCS bucket following this documentation

The StarRocks FE and CN yaml are shown below:

# This manifest deploys a StarRocks cluster running in shared data mode.
# see https://docs.starrocks.io/docs/cover_pages/shared_data_deployment/ for more information about shared-data mode.
#
# You will have to download and edit this YAML file to specify the details for your shared storage. See the 
# examples in the docs, and add your customizations to the ConfigMap `starrockscluster-sample-fe-cm` at the 
# bottom of this file.
# https://docs.starrocks.io/en-us/latest/deployment/deploy_shared_data#configure-fe-nodes-for-shared-data-starrocks

apiVersion: starrocks.com/v1
kind: StarRocksCluster
metadata:
  name: plaid-starrocks-cluster   # change the name if needed.
spec:
  starRocksFeSpec:
    image: starrocks/fe-ubuntu:latest
    replicas: 3
    limits:
      memory: 4Gi
    requests:
      cpu: '1'
      memory: 4Gi
    configMapInfo:
      configMapName: plaid-starrockscluster-fe-cm
      resolveKey: fe.conf
  starRocksCnSpec:
    image: starrocks/cn-ubuntu:latest
    replicas: 1
    limits:
      memory: 4Gi
    requests:
      cpu: '1'
      memory: 4Gi
    autoScalingPolicy: # Automatic scaling policy of the CN cluster.
      maxReplicas: 10 # The maximum number of CNs is set to 10.
      minReplicas: 1 # The minimum number of CNs is set to 1.
      # operator creates an HPA resource based on the following field.
      # see https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ for more information.
      hpaPolicy:
        metrics: # Resource metrics
          - type: Resource
            resource:
              name: memory  # The average memory usage of CNs is specified as a resource metric.
              target:
                # The elastic scaling threshold is 60%.
                # When the average memory utilization of CNs exceeds 60%, the number of CNs increases for scale-out.
                # When the average memory utilization of CNs is below 60%, the number of CNs decreases for scale-in.
                averageUtilization: 60
                type: Utilization
          - type: Resource
            resource:
              name: cpu # The average CPU utilization of CNs is specified as a resource metric.
              target:
                # The elastic scaling threshold is 60%.
                # When the average CPU utilization of CNs exceeds 60%, the number of CNs increases for scale-out.
                # When the average CPU utilization of CNs is below 60%, the number of CNs decreases for scale-in.
                averageUtilization: 60
                type: Utilization
        behavior: #  The scaling behavior is customized according to business scenarios, helping you achieve rapid or slow scaling or disable scaling.
          scaleUp:
            policies:
              - type: Pods
                value: 1
                periodSeconds: 10
          scaleDown:
            selectPolicy: Disabled

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: plaid-starrockscluster-fe-cm
  labels:
    cluster: starrockscluster-plaid-cp
data:
  fe.conf: |
    LOG_DIR = ${STARROCKS_HOME}/log
    DATE = "$(date +%Y%m%d-%H%M%S)"
    JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:${LOG_DIR}/fe.gc.log.$DATE"
    JAVA_OPTS_FOR_JDK_9="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
    JAVA_OPTS_FOR_JDK_11="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
    http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
    mysql_service_nio_enabled = true
    sys_log_level = INFO
    # config for shared-data mode
    run_mode = shared_data
    cloud_native_meta_port = 6090
    # Whether volume can be created from conf. If it is enabled, a builtin storage volume may be created.
    enable_load_volume_from_conf = true
    
    # GCS uses S3 Protocol
    cloud_native_storage_type = S3

    # For example, testbucket/subpath
    aws_s3_path = plaidcloud-starrocks-test/ci/plaid-cp

    # For example: us-east1
    aws_s3_region = us-central1

    # For example: https://storage.googleapis.com
    aws_s3_endpoint = https://storage.googleapis.com

    aws_s3_access_key = XXXXXXX
    aws_s3_secret_key = XXXXXXX

Once the cluster starts and there are 3 FE and 1 CN pods running, attempt to create the databases and tables as shown in this documentation

Everything seems to work correctly including creation of the database but when the table creation process is attempted, the following error is produced:

ERROR 1064 (HY000): Unexpected exception: fail to create tablet: 10011: [create tablet failed]

I cannot seem to find any additional information on that error and the logs do not have any additional depth to indicate why the creation process is failing.

Additional information:

I confirmed that there is a default storage volume configured. This was the automatically created one at cluster start.

mysql> describe storage volume builtin_storage_volume\g
+------------------------+------+-----------+--------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| Name                   | Type | IsDefault | Location                                   | Params                                                                                                                                                                                                                           | Enabled | Comment |
+------------------------+------+-----------+--------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| builtin_storage_volume | S3   | true      | s3://plaidcloud-starrocks-test/ci/plaid-cp | {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"https://storage.googleapis.com","aws.s3.region":"us-central1","aws.s3.use_instance_profile":"false","aws.s3.use_aws_sdk_default_behavior":"false"} | true    |         |
+------------------------+------+-----------+--------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
1 row in set (0.02 sec)

The query is the one from the example but the storage volume has been changed to builtin_storage_volume

mysql> CREATE TABLE IF NOT EXISTS detail_demo (
    ->     recruit_date  DATE           NOT NULL COMMENT "YYYY-MM-DD",
    ->     region_num    TINYINT        COMMENT "range [-128, 127]",
    ->     num_plate     SMALLINT       COMMENT "range [-32768, 32767] ",
    ->     tel           INT            COMMENT "range [-2147483648, 2147483647]",
    ->     id            BIGINT         COMMENT "range [-2^63 + 1 ~ 2^63 - 1]",
    ->     password      LARGEINT       COMMENT "range [-2^127 + 1 ~ 2^127 - 1]",
    ->     name          CHAR(20)       NOT NULL COMMENT "range char(m),m in (1-255) ",
    ->     profile       VARCHAR(500)   NOT NULL COMMENT "upper limit value 65533 bytes",
    ->     ispass        BOOLEAN        COMMENT "true/false")
    -> DUPLICATE KEY(recruit_date, region_num)
    -> DISTRIBUTED BY HASH(recruit_date, region_num)
    -> PROPERTIES (
    ->     "storage_volume" = "builtin_storage_volume",
    ->     "datacache.enable" = "true",
    ->     "datacache.partition_duration" = "1 MONTH",
    ->     "enable_async_write_back" = "false"
    -> );
ERROR 1064 (HY000): Unexpected exception: fail to create tablet: 10011: [create tablet failed]
mysql> 

I tried prefixing the storage path with gs:// but the resulting path that was created had a double prefix ofs3://gs://. I removed the gs://. I also tried the storage path with and without a trailing / but it didn't seem to have an impact.

Expected behavior

The table is created and data can be inserted/loaded into the table.

Please complete the following information

• Operator Version: starrocks/operator:v1.8.7
• Chart Name: kube-starrocks
• Chart Version v1.8.7

[kevincai]

the location is specified with s3:// is the correct one, we are taking google cloud storage as a variant of s3-compatible storage.

[inviscid]

Thanks for confirming. I put it back and the path is now just prefixed with s3://. Any idea why I might be getting that failed to create tablet error?

[inviscid]

I'd be happy to try a different example if there is a better one for the shared storage approach. I'm trying to get a functioning StarRocks cluster in place to test performance with our data and usage patterns. Any suggestions welcome. Thx.

[kevincai]

can you have a try with both BE and CN enabled in the cluster, I doubt there is a bug that causes the creation failure if no BE there.

[kevincai]

that is, add the following snippet into your cr yaml

  starRocksBeSpec:
    image: starrocks/be-ubuntu:latest
    replicas: 1
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      cpu: 1
      memory: 8Gi

[kevincai]

Since we use ArgoCD for deployments, the operator is NOT installed in the starrocks namespace but rather in cluster-components namespace. It seems to start up and the starrocks clusters deploy so I hope this is not an issue.

This is fine.

GCS bucket using S3 protocol - Verified the credentials allow full control of bucket

Yes, it works.

GCS bucket is empty - we have a path specified in the config. This path does not exist yet since the bucket is empty. The example has a minor typo which was easily fixed enable_load_volume_from_conf = ture to true

I see, thanks for pointing out!

[inviscid]

Deployed the BE node but still getting the same error. Confirmed there is nothing additional in the FE, CN, or BE logs to indicate why the tablet creation failed.

ERROR 1064 (HY000): Unexpected exception: fail to create tablet: 10004: [create tablet failed]

Do the FE nodes need to have PVs so they can maintain metadata persistence?

[inviscid]

If it would be helpful, I would be happy to create a K8s cluster and deploy SR where we could work together on it. I can't get you access to the current deployment but it wouldn't take much effort to create a separate cluster in a GCP project you could have access to.

[inviscid]

With further testing it looks like I can finally get this to work. However, I think there might be something wrong with the initial builtin_storage_volume setup.

I cannot use the builtin_storage_volume. It has identical parameters specified in the FE config but I consistently get the tablet create failure. When I create a fresh storage volume using the same parameters and then use that newly created storage volume, I can create the table.

I know you can't see the credentials in this view but I have manually verified they are identical

mysql> describe storage volume builtin_storage_volume;
+------------------------+------+-----------+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| Name                   | Type | IsDefault | Location                                       | Params                                                                                                                                                                                                                           | Enabled | Comment |
+------------------------+------+-----------+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| builtin_storage_volume | S3   | false     | s3://plaidcloud-starrocks-testing/ci/plaid-cp/ | {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"https://storage.googleapis.com","aws.s3.region":"us-central1","aws.s3.use_instance_profile":"false","aws.s3.use_aws_sdk_default_behavior":"false"} | true    |         |
+------------------------+------+-----------+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
1 row in set (0.02 sec)

mysql> describe storage volume gcs_volume;
+------------+------+-----------+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| Name       | Type | IsDefault | Location                                       | Params                                                                                                                                                                                                                           | Enabled | Comment |
+------------+------+-----------+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| gcs_volume | S3   | true      | s3://plaidcloud-starrocks-testing/ci/plaid-cp/ | {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"https://storage.googleapis.com","aws.s3.region":"us-central1","aws.s3.use_instance_profile":"false","aws.s3.use_aws_sdk_default_behavior":"false"} | true    |         |
+------------+------+-----------+------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
1 row in set (0.02 sec)

Trying table creation with the builtin_storage_volume results in:

ERROR 1064 (HY000): Unexpected exception: fail to create tablet: 10004: [create tablet failed]

And simply switching the storage volume to gcs_volume returns the following on table creation, as expected:

mysql> CREATE TABLE IF NOT EXISTS detail_demo (
    ->     recruit_date  DATE           NOT NULL COMMENT "YYYY-MM-DD",
    ->     region_num    TINYINT        COMMENT "range [-128, 127]",
    ->     num_plate     SMALLINT       COMMENT "range [-32768, 32767] ",
    ->     tel           INT            COMMENT "range [-2147483648, 2147483647]",
    ->     id            BIGINT         COMMENT "range [-2^63 + 1 ~ 2^63 - 1]",
    ->     password      LARGEINT       COMMENT "range [-2^127 + 1 ~ 2^127 - 1]",
    ->     name          CHAR(20)       NOT NULL COMMENT "range char(m),m in (1-255) ",
    ->     profile       VARCHAR(500)   NOT NULL COMMENT "upper limit value 65533 bytes",
    ->     ispass        BOOLEAN        COMMENT "true/false")
    -> DUPLICATE KEY(recruit_date, region_num)
    -> DISTRIBUTED BY HASH(recruit_date, region_num)
    -> PROPERTIES (
    ->     "storage_volume" = "gcs_volume",
    ->     "datacache.enable" = "true",
    ->     "datacache.partition_duration" = "1 MONTH",
    ->     "enable_async_write_back" = "false"
    -> );
Query OK, 0 rows affected (0.35 sec)

I think there might be something strange going on with the initial builtin_storage_volume creation. Is it possible this persists even after tearing down the SR instance and might have old data? I can't find a secret or anything else that persists though.

[DanRoscigno]

@inviscid

Hi Paul, I was working on docs for this today and saw your issue. With version 3.2.7 of StarRocks the builtin storage is working (I did not test older versions). Here is my edit of your config. All I changed were:

• the CPU/memory specs to fit in my small k8s env
• the GCS details
• My GCS path only has two levels, yours has three (plaidcloud-starrocks-test/ci/plaid-cp). I wonder if that was a problem?

As soon as I created a database and table in the SQL client a folder was added to the bucket and populated with metadata.

#This manifest deploys a StarRocks cluster running in shared data mode.
# see https://docs.starrocks.io/docs/cover_pages/shared_data_deployment/ for more information about shared-data mode.
#
# You will have to download and edit this YAML file to specify the details for your shared storage. See the
# examples in the docs, and add your customizations to the ConfigMap `starrockscluster-sample-fe-cm` at the
# bottom of this file.
# https://docs.starrocks.io/en-us/latest/deployment/deploy_shared_data#configure-fe-nodes-for-shared-data-starrocks

apiVersion: starrocks.com/v1
kind: StarRocksCluster
metadata:
  name: plaid-starrocks-cluster   # change the name if needed.
spec:
  starRocksFeSpec:
    image: starrocks/fe-ubuntu:3.2.7
    replicas: 3
    limits:
      memory: 3Gi
    requests:
      cpu: '1'
      memory: 1Gi
    configMapInfo:
      configMapName: plaid-starrockscluster-fe-cm
      resolveKey: fe.conf
  starRocksCnSpec:
    image: starrocks/cn-ubuntu:3.2.7
    replicas: 1
    limits:
      memory: 5Gi
    requests:
      cpu: '1'
      memory: 4Gi
    autoScalingPolicy: # Automatic scaling policy of the CN cluster.
      maxReplicas: 10 # The maximum number of CNs is set to 10.
      minReplicas: 1 # The minimum number of CNs is set to 1.
      # operator creates an HPA resource based on the following field.
      # see https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ for more information.
      hpaPolicy:
        metrics: # Resource metrics
          - type: Resource
            resource:
              name: memory  # The average memory usage of CNs is specified as a resource metric.
              target:
                # The elastic scaling threshold is 60%.
                # When the average memory utilization of CNs exceeds 60%, the number of CNs increases for scale-out.
                # When the average memory utilization of CNs is below 60%, the number of CNs decreases for scale-in.
                averageUtilization: 60
                type: Utilization
          - type: Resource
            resource:
              name: cpu # The average CPU utilization of CNs is specified as a resource metric.
              target:
                # The elastic scaling threshold is 60%.
                # When the average CPU utilization of CNs exceeds 60%, the number of CNs increases for scale-out.
                # When the average CPU utilization of CNs is below 60%, the number of CNs decreases for scale-in.
                averageUtilization: 60
                type: Utilization
        behavior: #  The scaling behavior is customized according to business scenarios, helping you achieve rapid or slow scaling or disable scaling.
          scaleUp:
            policies:
              - type: Pods
                value: 1
                periodSeconds: 10
          scaleDown:
            selectPolicy: Disabled

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: plaid-starrockscluster-fe-cm
  labels:
    cluster: starrockscluster-plaid-cp
data:
  fe.conf: |
    LOG_DIR = ${STARROCKS_HOME}/log
    DATE = "$(date +%Y%m%d-%H%M%S)"
    JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:${LOG_DIR}/fe.gc.log.$DATE"
    JAVA_OPTS_FOR_JDK_9="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
    JAVA_OPTS_FOR_JDK_11="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
    http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
    mysql_service_nio_enabled = true
    sys_log_level = INFO
    # config for shared-data mode
    run_mode = shared_data
    cloud_native_meta_port = 6090
    # Whether volume can be created from conf. If it is enabled, a builtin storage volume may be created.
    enable_load_volume_from_conf = true

    # GCS uses S3 Protocol
    cloud_native_storage_type = S3

    # For example, testbucket/subpath
    aws_s3_path = shared-data-qs/starrocks-dev

    # For example: us-east1
    aws_s3_region = northamerica-northeast1

    # For example: https://storage.googleapis.com
    aws_s3_endpoint = https://storage.googleapis.com
    
    aws_s3_access_key = GO...
    aws_s3_secret_key = iE...