gcpdiag
gcpdiag copied to clipboard
Intermittent warning/skips on Composer rules
This was working fine on 0.59, now there seems to be some lock issue on Composer API queries. Here's the command used:
gcpdiag lint --project=my-project --include=composer
At the start of the command, the following log will be temporarily visible on the console, before the results are printed.
... still fetching logs (project: my-project, resource type: cloud_composer_environment, max wait: 117s)
After two minutes, some of the checks are skipped because of (I guess) a timeout:
🔎 composer/BP/2023_001: Cloud Composer logging level is set to INFO
- my-project/europe-west1/my-composer [ OK ]
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/BP/2023_002
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/BP/2023_003
🔎 composer/ERR/2022_001: Composer Service Agent permissions
- my-project [ OK ]
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/ERR/2022_002
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/ERR/2023_001
🔎 composer/WARN/2022_001: Composer Service Agent permissions for Composer 2.x
- my-project [ OK ]
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2022_002
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2022_003
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_001
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_002
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_003
🔎 composer/WARN/2023_004: Cloud Composer database CPU usage does not exceed 80%
- my-project/europe-west1/my-composer [ OK ]
🔎 composer/WARN/2023_005: Cloud Composer is consistently in healthy state
- my-project/europe-west1/my-composer [ OK ]
🔎 composer/WARN/2023_006: Airflow schedulers are healthy for the last hour
- my-project/europe-west1/my-composer [ OK ]
🔎 composer/WARN/2023_007: Cloud Composer Scheduler CPU limit exceeded.
- my-project/europe-west1/my-composer [ OK ]
🔎 composer/WARN/2023_008: Cloud Composer Airflow database is in healthy state
- my-project/europe-west1/my-composer [ OK ]
Rules summary: 9 skipped, 8 ok, 0 failed
If I launch the command with a more restrictive parameter, like --include=composer/WARN, this still happens although less frequently.
Thanks for the issue report. Can you check if this still occurs with v0.63 recently released?