InvalidSubnet when auto-rds-s3-setup.py script is used to install Kubeflow

[dsskonuru]

Hi Everyone, I am installing Kubeflow for the first time in an AWS environment. I am following the documentation I get the following error when I run the auto-rds-s3-setup.py script.

=================================================================
                            RDS Setup                            
=================================================================
Creating DB subnet group...
Traceback (most recent call last):
  File ".../kubeflow-manifests/tests/e2e/utils/rds-s3/auto-rds-s3-setup.py", line 690, in <module>
    main()
  File ".../kubeflow-manifests/tests/e2e/utils/rds-s3/auto-rds-s3-setup.py", line 44, in main
    setup_rds(rds_client, secrets_manager_client, eks_client, ec2_client)
  File ".../kubeflow-manifests/tests/e2e/utils/rds-s3/auto-rds-s3-setup.py", line 193, in setup_rds
    db_root_password = setup_db_instance(
                       ^^^^^^^^^^^^^^^^^^
  File ".../kubeflow-manifests/tests/e2e/utils/rds-s3/auto-rds-s3-setup.py", line 216, in setup_db_instance
    setup_db_subnet_group(rds_client, eks_client, ec2_client)
  File ".../kubeflow-manifests/tests/e2e/utils/rds-s3/auto-rds-s3-setup.py", line 224, in setup_db_subnet_group
    create_db_subnet_group(rds_client, eks_client, ec2_client)
  File ".../kubeflow-manifests/tests/e2e/utils/rds-s3/auto-rds-s3-setup.py", line 244, in create_db_subnet_group
    rds_client.create_db_subnet_group(
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidSubnet: An error occurred (InvalidSubnet) when calling the CreateDBSubnetGroup operation: Subnet IDs are required.

I was thinking the script should create all the required resources including subnets or allow me to provide the subnet IDs if I had to create them manually. Any insight into this is appreciated. Thanks!

[dsskonuru]

I figure out the issue. The code only checks for subnets created for eks-generated and cdk-generated clusters. I generated my cluster using Terraform

[ryansteakley]

@dsskonuru When you say clusters generated using terraform, did you use our terraform deployment option?

[dsskonuru]

I used the official EKS module (https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest) to build my EKS cluster

[ryansteakley]

@dsskonuru We will look into adding support for terraform generated clusters in the auto-setup script

[surajkota]

@dsskonuru would you like to contribute to this feature?

[timothyepping]

Running into the same issue when trying to deploy RDS and S3 with auto-rds-s3-setup.py using manifest deployment method.

                        RDS Setup

================================================================= Creating DB subnet group... Traceback (most recent call last): File "utils/rds-s3/auto-rds-s3-setup.py", line 892, in main() File "utils/rds-s3/auto-rds-s3-setup.py", line 45, in main setup_rds(rds_client, secrets_manager_client, eks_client, ec2_client) File "utils/rds-s3/auto-rds-s3-setup.py", line 311, in setup_rds db_root_password = setup_db_instance( File "utils/rds-s3/auto-rds-s3-setup.py", line 336, in setup_db_instance setup_db_subnet_group(rds_client, eks_client, ec2_client) File "utils/rds-s3/auto-rds-s3-setup.py", line 344, in setup_db_subnet_group create_db_subnet_group(rds_client, eks_client, ec2_client) File "utils/rds-s3/auto-rds-s3-setup.py", line 364, in create_db_subnet_group rds_client.create_db_subnet_group( File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 530, in _api_call return self._make_api_call(operation_name, kwargs) File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 964, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.InvalidSubnet: An error occurred (InvalidSubnet) when calling the CreateDBSubnetGroup operation: Subnet IDs are required.