KhronosGroup
Is a drastic reduction in number of tests from vulkan-cts-1.3.7.3 to vulkan-cts-1.3.8.0 expected and intentional?
vulkan-cts-1.3.8.0 seems to have only about two-thirds the number of tests that vulkan-cts-1.3.7.3 has.
From a recent comparison running tests on MoltenVK:
dEQP Core vulkan-cts-1.3.7.3-0-gd71a36db16d98313c431829432a136dbda692a08 (0xd71a36db) starting..
Test run totals:
Passed: 404332/4006625 (10.1%)
Failed: 11140/4006625 (0.3%)
Not supported: 3591092/4006625 (89.6%)
Warnings: 61/4006625 (0.0%)
Waived: 0/4006625 (0.0%)
dEQP Core vulkan-cts-1.3.8.0-0-gd13582ab1e753a41b2cedf8e3c504e1cb92edfe5 (0xd13582ab) starting..
Test run totals:
Passed: 392275/2702883 (14.5%)
Failed: 9919/2702883 (0.4%)
Not supported: 2300634/2702883 (85.1%)
Warnings: 55/2702883 (0.0%)
Waived: 0/2702883 (0.0%)
We seem to have an overall reduction of 1.3 M tests.
I found these comments in the vulkan-cts-1.3.8.0 release notes:
Remove duplicate pipeline protected access tests
Which seems to account for about 320 K tests.
Simplify BorderSwizzle TestNode
Which seems to account for about 280 K tests.
Reduce the amount of variations in
VK_EXT_host_image_copytests
Which seems to account for about 108 K tests.
Remove duplicate shader object protected image tests
Which seems to account for about 861 K tests.
There seems to be a large reduction in dEQP-VK.pipeline.*.extended_dynamic_state.* tests, but I can't see a clear pattern to those removed tests, nor do a see a clear note about this in the release notes.
These seem to account for about 145 K tests.
But this is all assuming that the test reductions are driven by these notes. I would have expected such a large change in overall number to be mentioned in the release notes, since the trend has always been to increase the test count from release to release.
So, just to confirm, is this overall drastic reduction correct, and intentional? On a full run, should I expect "only" 2.7M tests? Or is it possible that I have somehow pulled or built CTS in a manner that has lost a large number of tests?
Sounds about right. In addition to what you listed 1fa0038d687544faf9696505b90688e4ebc4e83e removed about 1.09m (1088703) tests faf4fbbc8f8e2ae777a679965e9fb79a1701ff42 removed about 139k (138866) tests
