KfAwsPlugin "managedRelationDatabase" does not work in KF 1.2.0

[karlschriek]

I spent some time today rolling out KF 1.2.0 on EKS using exclusively RDS databases instead of an on-cluster MySQL deployment.

The latest kfctl has a "managedRelationDatabase" field that can be set in the kfdef, such as here:

  plugins:
    - kind: KfAwsPlugin
      metadata:
        name: aws
      spec:
        auth:
          cognito:
            certArn: REPLACE_WITH_CERT_ARN
            cognitoAppClientId: REPLACE_WITH_COGNITO_APP_CLIENT_ID
            cognitoUserPoolArn: REPLACE_WITH_COGNITO_USER_POOL
            cognitoUserPoolDomain: REPLACE_WITH_COGNITO_DOMAIN
        region: REPLACE_WITH_AWS_REGION
        enablePodIamPolicy: true
        roles:
        - REPLACE_WITH_ROLE_NAME
        managedRelationDatabase:
          host: REPLACE_WITH_METADATA_RDS_ENDPOINT
          database: REPLACE_WITH_METADATA_RDS_DATABASE
          username: REPLACE_WITH_METADATA_DB_USERNAME
          password: REPLACE_WITH_METADATA_DB_PASSWORD
          port: REPLACE_WITH_METADATA_DB_PORT

This appears to be meant as a way to easily set the required overlays and ensure that the "external-mysql" overlays are rolled out instead of the default "db" ones. However, this did not work as it seems to update configmaps "metadata-config" and "api-service-config" that do not get rolled out at all and are not referenced by any of the points.

So I went about it manually. The following manually changes fixed it for me:

kfctl build -V -f kfctl_aws_cognito.1.2.0.yaml
sh ./configure_rds_overlays.sh
kfctl apply -V -f kfctl_aws_cognito.1.2.0.yaml

where configure_rds_overlays.sh has the following contents:

cat <<EoF > ${KF_CONFIGS_DIR}/.cache/manifests/manifests-1.2.0/pipeline/installs/generic/params.env
bucketName=mlpipeline
dbHost=${RDS_ENDPOINT}
dbPort=${RDS_PORT}
mlmdDb=${METADATA_DB_NAME}
cacheDb=${CACHE_DB_NAME}
pipelineDb=${PIPELINES_DB_NAME}
EoF

cat <<EoF > ${KF_CONFIGS_DIR}/.cache/manifests/manifests-1.2.0/pipeline/installs/generic/params-db-secret.env
username=${RDS_USERNAME}
password=${RDS_PASSWORD}
EoF

cat <<EoF > ${KF_CONFIGS_DIR}/.cache/manifests/manifests-1.2.0/metadata/overlays/external-mysql/params.env
MYSQL_HOST=${RDS_ENDPOINT}
MYSQL_DATABASE=${METADATA_DB_NAME}
MYSQL_PORT=${RDS_PORT}
MYSQL_ALLOW_EMPTY_PASSWORD=true
EoF

cat <<EoF > ${KF_CONFIGS_DIR}/.cache/manifests/manifests-1.2.0/metadata/overlays/external-mysql/secrets.env
MYSQL_USERNAME=${RDS_USERNAME}
MYSQL_PASSWORD=${RDS_PASSWORD}
EoF

cat <<EoF > ${KF_CONFIGS_DIR}/.cache/manifests/manifests-1.2.0/metadata/v3/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../overlays/external-mysql
- ../overlays/istio/virtual-service-metadata-grpc.yaml
configurations:
- params.yaml
EoF

It seems that the above (or at least something similar) is the behaviour that setting managedRelationDatabase is intended to invoke.

[karlschriek]

@PatrickXYS, @Jeffwan

[PatrickXYS]

        managedRelationDatabase:
          host: REPLACE_WITH_METADATA_RDS_ENDPOINT
          database: REPLACE_WITH_METADATA_RDS_DATABASE
          username: REPLACE_WITH_METADATA_DB_USERNAME
          password: REPLACE_WITH_METADATA_DB_PASSWORD
          port: REPLACE_WITH_METADATA_DB_PORT

I don't really think this will work, because those are interfaces we pre-defined, but not yet implement

[karlschriek]

Yes, that is the conclusion I came to as well. Doesn't look like it would be too hard to implement though? Essentially instead of creating the config maps as it currently does it does needs to adjust the overlays as described above. That would be quite useful since one then only needs to update the kfdef

[PatrickXYS]

All the other parts of implementation are finished, except how to pass password to replace K8s secret, it's fairly easy in the k8s onfigmap, however not in k8s secret.

It's about kfctl implementation, we generate all the manifests (including secrets and configmap) at first, then place them with our KfAwsPlugin parameter.

I touched it previously, might need to spend some time figuring it out.