NAS-135198 / 26.04 / HA CI tests for NVMe-oF(TCP)

[bmeagherix]

Add a suite of failover tests for NVMe-oF(TCP) - although only a subset of them will be run by default

test_failover: Basic failover tests across multiple dimensions

• Target implementation: kernel vs SPDK
• Failover mode: ANA vs IP takeover
• Failure type: orderly vs crash
• I/O state: active vs idle
• Namespace count: 1 vs 3

test_failover_scale: Large-scale failover tests

• Target implementation: kernel vs SPDK
• Failover mode: ANA vs IP takeover
• Failure type: orderly vs crash
• Scale: 51 subsystems (50 single-namespace + 1 with 20 namespaces) = 70 total namespaces
• Verifies unique data patterns per namespace survive failover
• Measures failover timing (must complete within MAX_FAILOVER_TIME seconds)

test_failover_and_failback: Failover + failback cycle tests

• Target implementation: kernel vs SPDK
• Failover mode: ANA vs IP takeover
• Pattern: crash failover -> orderly failback
• Verifies data integrity survives complete cycle back to original node
• Measures failover and failback timing (must complete within MAX_FAILOVER_TIME seconds)

Full Matrix (with RUN_FULL_MATRIX=1 - 44 tests total):

• TestFailover: 32 tests (2×2×2×2×2)
• TestFailoverScale: 8 tests (2×2×2)
• TestFailback: 4 tests (2×2)

Quick Subset (default - 10 tests total):

TestFailover (4 tests):

• Parameters: namespace_count=1, failure_type='orderly', io_active=True
• Combinations:
  • kernel + ANA
  • kernel + ip_takeover
  • spdk + ANA
  • spdk + ip_takeover

TestFailoverScale (2 tests):

• Specific combinations only:
  • kernel + ANA + orderly
  • spdk + ip_takeover + crash

TestFailback (4 tests - no filtering):

• All combinations run:
  • kernel + ANA
  • kernel + ip_takeover
  • spdk + ANA
  • spdk + ip_takeover

[bugclerk]

Jira URL: https://ixsystems.atlassian.net/browse/NAS-135198