refactor(gcloud): Introduce a test framework for Critical User Journeys

[cjac]

refactor(gcloud): Introduce a test framework for Critical User Journeys

This commit refactors the project's testing and automation scripts, moving them from a monolithic bin/ directory into a structured framework centered around Critical User Journeys (CUJs). The goal is to create a more maintainable and scalable way to exercise the use cases that are essential for our customers.

This new structure is designed for better organization, code reuse, and CI/CD integration.

Key changes include:

• CUJ Directory Structure:

  • gcloud/cuj/: Contains individual journey scripts. The first CUJ, standard-cluster-management, is introduced to handle the creation, deletion, and rebuilding of a standard Dataproc cluster.

  • gcloud/onboarding/: Holds scripts for setting up shared, prerequisite infrastructure (e.g., GCS buckets) that CUJs depend on.

  • gcloud/lib/: A new directory for shared bash libraries. common.sh is introduced to centralize functions for configuration loading, network management, and logging.

  • gcloud/ci/: Includes scripts specifically for CI/CD pipelines.

• CI Environment Management:

  • A new pristine_check.sh script is added to ensure the GCP project is in a clean state before a test run. It can operate in a strict validation mode (fail if resources exist) or a cleanup mode (actively delete resources).

• Configuration and Code Improvements:

  • Configuration is centralized and simplified in env.json, which is now loaded by the load_config function in common.sh.

  • Scripts are refactored to be more robust, with clear up, down, and validate commands, and checks for pre-existing resources.

  • The .gitignore is updated to exclude temporary files and local configuration.

This change lays the foundation for adding more complex user journeys, ensuring our core services are consistently tested from a customer perspective.

Resolves: https://github.com/GoogleCloudDataproc/initialization-actions/issues/1281 Relates: go/sf/59134938

[cjac]

This PR cannot resolve https://github.com/GoogleCloudDataproc/initialization-actions/issues/1281 until it implements the proxy-only CUJ.