macaron icon indicating copy to clipboard operation
macaron copied to clipboard

Unit tests in tests/slsa_analyzer/build_tool/test_gradle.py rely on network

Open tromai opened this issue 1 year ago • 1 comments

This applies for unit tests in tests/slsa_analyzer/build_tool/test_gradle.py that test the method get_group_ids https://github.com/oracle/macaron/blob/5522ec103d586789ec1a99db29c02d94a7949656/src/macaron/slsa_analyzer/build_tool/gradle.py#L170

This method eventually invokes get_group_id method that creates a subprocess to call gradlew shipped with Macaron or the gradlew within the target repository source code. https://github.com/oracle/macaron/blob/5522ec103d586789ec1a99db29c02d94a7949656/src/macaron/slsa_analyzer/build_tool/gradle.py#L231-L237

gradlew however, on first run would try to download the gradle distribution from the internet to bootstrap it. If it cannot download the gradle distribution, the relevant unit tests will fail.

To reproduce:

# Run within the Macaron repository. Make sure to turn off your internet connection.
mv ~/.gradle ~/.gradle_bak
make setup
pytest tests/slsa_analyzer/build_tool/test_gradle.py

The error:

FAILED tests/slsa_analyzer/build_tool/test_gradle.py::test_get_group_ids_nested_projects - AssertionError: assert set() == {'io.micronaut'}
  
  Extra items in the right set:
  'io.micronaut'
  
  Full diff:
  + set()
  - {
  -     'io.micronaut',
  - }
FAILED tests/slsa_analyzer/build_tool/test_gradle.py::test_get_group_ids_separate_projects - AssertionError: assert set() == {'io.micronaut.foo', 'io.micronaut.bar'}
  
  Extra items in the right set:
  'io.micronaut.foo'
  'io.micronaut.bar'
  
  Full diff:
  + set()
  - {
  -     'io.micronaut.bar',
  -     'io.micronaut.foo',
  - }
FAILED tests/slsa_analyzer/build_tool/test_gradle.py::test_get_group_ids_timeout[invalid-expected1] - AssertionError: assert set() == {'io.micronaut'}
  
  Extra items in the right set:
  'io.micronaut'
  
  Full diff:
  + set()
  - {
  -     'io.micronaut',
  - }
==================== 3 failed, 26 passed in 42.13s ====================

I wonder if we should look into fixing it somehow? Or it's okay to leave it as it is.

tromai avatar Aug 15 '24 05:08 tromai

After some discussion, we agreed that we should remove the usage of gradlew to obtain the group id and remove relevant tests. This is because this logic is not being used anywhere in Macaron.

tromai avatar Aug 15 '24 06:08 tromai