xharness icon indicating copy to clipboard operation
xharness copied to clipboard

[android] Add support for running workitems on specified Android emulator API levels

Open Copilot opened this issue 5 months ago • 3 comments

This PR adds support for specifying multiple Android API levels when running tests in XHarness, addressing the need for the new Helix "multi" queue that contains pre-installed emulators covering all API levels.

Changes Made

Enhanced --api-version Argument

Modified the existing ApiVersionArgument to support multiple values by extending RepeatableArgument instead of OptionalIntArgument:

# Run tests on multiple API levels
xharness android-headless test --api-version=28 --api-version=29 --api-version=30 ...

# Existing single API level support is preserved
xharness android-headless test --api-version=28 ...

The argument can now be specified multiple times to test across different API levels, with the help text automatically indicating "Can be used more than once".

Emulator Management

Extended AdbRunner with new emulator management capabilities:

  • StartEmulator(int apiLevel) - Starts an emulator for a specific API level
  • StopEmulator(int apiLevel) - Stops an emulator for a specific API level
  • StopEmulators(IEnumerable<int> apiLevels) - Stops multiple emulators

The system automatically starts emulators for specified API levels when using multiple --api-version arguments, addressing the requirement that the "multi" queue does not start emulators by default.

Distinguishable Results

When running tests on multiple API levels, results are stored in separate directories:

  • /output/api-28/ for API level 28 results
  • /output/api-29/ for API level 29 results
  • etc.

This ensures results are distinguishable when the same workitem runs on multiple API levels.

Backward Compatibility

  • Added FirstApiVersion property to ApiVersionArgument for backward compatibility with existing commands (android test, android install, android device) that only support single API versions
  • The --api-version argument maintains full backward compatibility - when used once, it behaves exactly as before
  • API level range validation (16-35) is enforced for all values

Comprehensive Testing

Added unit tests covering:

  • Single and multiple API version argument parsing
  • API level range validation
  • Command argument validation

Usage Examples

# Test on multiple API levels with emulator auto-start
xharness android-headless test \
  --test-path=/path/to/test \
  --runtime-folder=/path/to/runtime \
  --test-assembly=MyTest.dll \
  --test-script=test.sh \
  --output-directory=/path/to/output \
  --api-version=28 --api-version=29 --api-version=30

# Results will be saved to:
# /path/to/output/api-28/
# /path/to/output/api-29/  
# /path/to/output/api-30/

Fixes #1459.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot avatar Sep 12 '25 13:09 Copilot

Instead of adding a new property for --api-levels , can we make --api-version work for the same propose, adding one or several ?

rmarinho avatar Sep 12 '25 15:09 rmarinho

@copilot Instead of adding a new property for --api-levels , can we make --api-version work for the same propose, adding one or several ?

kotlarmilos avatar Oct 06 '25 12:10 kotlarmilos

Instead of adding a new property for --api-levels , can we make --api-version work for the same propose, adding one or several ?

Done! The --api-version argument now supports multiple values. You can specify it multiple times like --api-version=28 --api-version=29 --api-version=30 to run tests on multiple API levels. The --api-levels argument has been removed. (commit f0280d5)

@copilot Instead of adding a new property for --api-levels , can we make --api-version work for the same propose, adding one or several ?

Done! The --api-version argument now supports multiple values. You can specify it multiple times like --api-version=28 --api-version=29 --api-version=30 to run tests on multiple API levels. The --api-levels argument has been removed. (commit f0280d5)

Copilot avatar Oct 06 '25 12:10 Copilot