Fix: Handle Deleted Namespaces Gracefully During Sync

[tricktron]

Summary

Fix infinite sync failure loops when managed namespaces are deleted by implementing automatic namespace validation and cleanup during cluster cache synchronization.

Fixes: https://github.com/argoproj/argo-cd/issues/24709

Problem

When namespaces managed by ArgoCD are deleted without first removing the managed-by label, the GitOps Engine enters an infinite failure loop during cluster cache sync operations. The processApi() function attempts to list resources in deleted namespaces, resulting in 403 Forbidden errors from the Kubernetes API. This causes:

• Complete sync failures every 10 minutes (default cache sync interval)
• ArgoCD becomes unresponsive until manual controller restart
• No automatic recovery mechanism exists

Root Cause: The sync() process iterates through c.namespaces slice containing deleted namespace names but has no validation to check if those namespaces still exist before attempting API operations.

Solution

Implement namespace validation with automatic cleanup:

Key Changes

1. namespaceExists() function - Validates namespace existence using canonical apierrors.IsNotFound() detection
2. Enhanced processApi() - Skip deleted namespaces during resource processing using thread-safe tracking
3. Post-sync cleanup in sync() - Remove deleted namespaces from configuration after parallel processing completes

I also added a test for the scenario called TestSyncWithDeletedNamespace and added the default namespace in other tests to not break them.

[codecov[bot]]

Codecov Report

:x: Patch coverage is 90.00000% with 4 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 47.63%. Comparing base (8849c3f) to head (0b5d0a1). :warning: Report is 62 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/cache/cluster.go	90.00%	3 Missing and 1 partial :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #785      +/-   ##
==========================================
- Coverage   54.26%   47.63%   -6.63%     
==========================================
  Files          64       64              
  Lines        6164     6627     +463     
==========================================
- Hits         3345     3157     -188     
- Misses       2549     3212     +663     
+ Partials      270      258      -12     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sonarqubecloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
27.9% Duplication on New Code

See analysis details on SonarQube Cloud