debug: Fix tests so they pass/not_applicable on HiFive1

[timsifive]

#485 introduces a list of tests that are known to cause an error on HiFive1. This issue is to go through that list, and fix the tests/infrastructure/whatever so that all these tests either pass, or return not_applicable.

[TommyMurphyTM1234]

Hi @timsifive (and/or others) - I'm a bit confused about the how exactly specific tests are categorised as "not_applicable". If I use the following (i.e. without appending --exclude-tests hifive1_excludes.yaml to specify the HiFive1 specific exclude list):

./gdbserver.py targets/SiFive/HiFive1.py

then I get this:

:::::::::::::::::::::::::::[ ran 72 tests in 355s ]:::::::::::::::::::::::::::
18 tests returned not_applicable
43 tests returned pass
5 tests returned fail
   EtriggerTest > logs/20231006-114621-HiFive1-EtriggerTest.log
   IcountTest > logs/20231006-114642-HiFive1-IcountTest.log
   InstantHaltTest > logs/20231006-114655-HiFive1-InstantHaltTest.log
   Semihosting > logs/20231006-114937-HiFive1-Semihosting.log
   SemihostingFileio > logs/20231006-114944-HiFive1-SemihostingFileio.log
6 tests returned exception
   Hwbp2 > logs/20231006-114632-HiFive1-Hwbp2.log
   ItriggerTest > logs/20231006-114713-HiFive1-ItriggerTest.log
   MemTestReadInvalid > logs/20231006-114826-HiFive1-MemTestReadInvalid.log
   MemorySampleMixed > logs/20231006-114833-HiFive1-MemorySampleMixed.log
   MemorySampleSingle > logs/20231006-114845-HiFive1-MemorySampleSingle.log
   RepeatReadTest > logs/20231006-114931-HiFive1-RepeatReadTest.log

What I don't understand is what dictates this:

18 tests returned not_applicable

I've skimmed the Python scripts but cannot answer this question so far. Can anybody else? Thanks.

[TommyMurphyTM1234]

Ah - I think I get it now - class Target sets certain attributes to default values that disable certain tests because they depend on specific target capabilities/features. Those tests are only enabled if a specific target derived from class Target (e.g. class HiFive1) overrides the relevant attributes in order to declare support for the tests/required capabities?

Specifically - the following tests are assumed to be "not_applicable" by default and the list links to the relevant class Target attributes and checks in gdbserver.py:

• CeaseStepiTest
  • gdbserver.py check: https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1864
  • class Target attribute: https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets.py#L54
• CustomRegisterTest
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L193
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets.py#L101
• FreeRtosTest
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1949
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets.py#L130
• HwbpManual
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets.py#L122
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L612
• MemTestBlockReadInvalid
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L298
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets.py#L104
• MulticoreRegTest
• MulticoreRtosSwitchActiveHartTest
• MulticoreRunAllHaltOne
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1170
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets/SiFive/HiFive1.py#L11
• PrivChange
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1607
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1574
• SmpSimultaneousRunHalt
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1227
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets/SiFive/HiFive1.py#L11
• StepThread2Test
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L2020
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets/SiFive/HiFive1.py#L11
• Sv32Test
• Sv39Test
• Sv48Test
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1639
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets/SiFive/HiFive1.py#L6
• UnavailableCycleTest
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1924
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets.py#L133
• UnavailableMultiTest
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1820
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets.py#L133
• UnavailableRunTest
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1887
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets.py#L54
• VectorTest
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/gdbserver.py#L1713
  • https://github.com/riscv-software-src/riscv-tests/blob/9905a434c9da6588763b31dd01d263144e0db0ed/debug/targets/SiFive/HiFive1.py#L8

[timsifive]

Right. The not_applicable() method is older than the HiFive1 exclusion list. In theory it's more flexible if we add support for more hardware in the future. In practice I haven't really kept it up-to-date. Ideally the tests/target definitions would be updated so that HiFive1 wouldn't need the separate exclusion list.