robottelo
robottelo copied to clipboard
SatelliteQE
Convert manifest-dependent API tests to manifester
This PR will convert all API tests that use manifests to obtain their manifests using manifester. I am planning on pushing one commit per module in an attempt to stay organized. The initial commit converts the api/test_contentview.py module. Note that, in this module, the TestContentViewRedHatOstreeContent class is decorated with @pytest.mark.skip_if_open("BZ:1625783"), but this BZ is closed WONTFIX. With the marker removed, these tests fail, but the failures are unrelated to the use of manifester.
trigger: test-robottelo pytest: tests/foreman/api/test_contentview.py::TestContentViewRedHatContent
A couple of the content management tests are failing locally during Capsule deployment, but I don't think manifester is the cause. The tests are passing in CI, though, so I'd like to see what results I get with PRT.
trigger: test-robottelo pytest: tests/foreman/api/test_contentmanagement.py -k "test_positive_sync_kickstart_repo or test_positive_sync_kickstart_check_os or test_positive_iso_library_sync"
trigger: test-robottelo pytest: tests/foreman/api/test_contentmanagement.py -k 'test_positive_sync_kickstart_repo or test_positive_sync_kickstart_check_os or test_positive_iso_library_sync'
The two failures in the most recent PRT run are not consistent with my local failures. In the PRT run, the two tests that failed both did so while attempting to sync the RHEL 8 BaseOS kickstart repo. Strangely, it looks to me like TestSatelliteContentManagement::test_positive_sync_kickstart_repo successfully synced the repo, set the download policy explicitly to immediate, and subsequently failed to sync the same repo. Looking at the Nailgun Repository entity [1], though, it looks to me like the default policy is immediate, so I'm not sure why the test is written in this way. If my reading of the Repository entity is correct, then TestSatelliteContentManagement::test_positive_sync_kickstart_check_os is also failing to sync the RHEL 8 BaseOS kickstart repo with the immediate download policy.
I will do some additional debugging tomorrow.
[1] https://github.com/SatelliteQE/nailgun/blob/53b6f11cc77eb202bbf2066dbb27c79b85f35e42/nailgun/entities.py#L6145
trigger: test-robottelo pytest: tests/foreman/api/test_contentmanagement.py -k 'test_positive_sync_kickstart_repo'
trigger: test-robottelo pytest: tests/foreman/api/test_contentmanagement.py -k 'test_positive_sync_kickstart_repo or test_positive_sync_kickstart_check_os or test_positive_iso_library_sync'
All the api/test_contentmanagement.py tests are passing today, so I am chalking up yesterday's failures to a transient CDN issue. Moving on to api/test_convert2rhel.py.
Just to clarify, this PR is still in progress. The following modules still need to be converted before this PR is taken out of draft: api/test_convert2rhel.py api/test_organization.py api/test_product.py api/test_reporttemplates.py api/test_repository.py api/test_repository_set.py api/test_errata.py api/test_syncplan.py api/test_subscription.py
Note: as mentioned in my last commit message, the tests in api/test_convert2rhel.py are failing both locally and in CI with the same error.
trigger: test-robottelo pytest: tests/foreman/api -k "test_negative_check_org_endpoint or test_positive_filter_product_list or test_positive_generate_entitlements_report or test_positive_schedule_entitlements_report"
trigger: test-robottelo pytest: tests/foreman/api/ -k 'test_negative_check_org_endpoint or test_positive_filter_product_list or test_positive_generate_entitlements_report or test_positive_schedule_entitlements_report'
The two report templates tests failed with:
2022-10-19 17:21:22 - nailgun.client - WARNING - Received HTTP 400 response: {"displayMessage":"The specified organization is in Simple Content Access mode. Attaching subscriptions is disabled","errors":["The specified organization is in Simple Content Access mode. Attaching subscriptions is disabled"]}
This is likely because the PRT run was initiated before https://github.com/SatelliteQE/robottelo/pull/10179 was merged. Running them again to be sure.
trigger: test-robottelo pytest: tests/foreman/api/ -k 'test_positive_generate_entitlements_report or test_positive_schedule_entitlements_report'
trigger: test-robottelo pytest: tests/foreman/api/test_repository.py -k 'test_positive_recreate_pulp_repositories or test_positive_sync_rh or test_positive_sync_repo_null_contents_changed'
The three tests in api/test_errata.py that use manifests are failing in CI, so I'm not going to run PRT for those tests.
trigger: test-robottelo pytest: tests/foreman/api/test_syncplan.py -k 'test_positive_synchronize_rh_product_future_sync_date or test_positive_synchronize_rh_product_past_sync_date'
trigger: test-robottelo pytest: tests/foreman/api/test_subscription.py::test_positive_subscription_status_disabled
Looking into the logs for the PRT run, it appears that the rhel6:latest base image is built on RHEL 6.9:
'[D 221021 22:05:42 container:197] {'id': 'f178b4887043966ecd44a4ffd72fdae9a863e389ca415dc64f53e995b7b5df28', 'image': 'sha256:46d6d65a8cf78763f03b43c62641e019312ebe3d99eaea30ef07b6a8ea94e4d0', 'name': 'satci_c9bdfdf1', 'container_config': {'Hostname': 'f178b4887043', *** 8< SNIP 8< *** 'JENKINS_URL=https://satqe-jenkins-csb-satellite-qe.apps.ocp-c1.prod.psi.redhat.com/job/robottelo-pr-testing/1389/', *** 8< SNIP 8< *** 'Image': 'rhel6:latest', 'Volumes': None, 'WorkingDir': '/root', 'Entrypoint': None, 'OnBuild': None, 'Labels': {'BROKER_ORIGIN': 'rhel_contenthost:/opt/app-root/src/robottelo/pytest_fixtures/core/contenthosts.py', 'BZComponent': 'rhel-server-docker', *** 8< SNIP 8< *** 'Name': 'rhel6', 'Version': '6.9', 'architecture': 'x86_64', 'authoritative-source-url': 'registry.access.redhat.com', 'broker_compatible': 'True', 'build-date': '2018-05-03T23:40:14.946673''
RHEL 6.10 is the actual latest RHEL 6 y-stream release, and I don't believe that Simple Content Access is supported on RHEL 6.9 (though the version of subscription-manager is really the deciding factor there). This seems to be the cause of the failure; sub-man does not recognize that SCA is enabled:
[D 221021 22:05:47 hosts:106] f178b4887043 executing command: subscription-manager register --org LVOoUnxoQ --activationkey KmUprsrtQR --force
[D 221021 22:05:50 hosts:108] f178b4887043 command result:
stdout:
The system has been registered with ID: 80f1eb34-ae74-426a-95e9-c8c1caa89b3a
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status: Not Subscribed
Unable to find available subscriptions for all your installed products.
stderr:
status: 1
If my analysis is correct, then to resolve this issue, we'll need to either use a more current RHEL 6 image or convince the maintainers of the RHEL 6 UBI to update the image that we're pulling now. Either way, manifester does not appear to be the cause of this failure, as far as I can tell.
Ok, some additional information on RHEL 6 and SCA: the first version of subscription-manager that is SCA-aware is 1.24 [1]. The latest version of subscription-manager to ship for RHEL 6 is 1.20. So api/test_subscription.py::test_positive_subscription_status_disabled is going to fail on RHEL 6 unless some sort of alternative logic is provided for that case. Consequently, I feel confident in stating that manifester is not the cause of that test failure.
[1] https://access.redhat.com/solutions/5060001
trigger: test-robottelo pytest: tests/foreman/api/test_subscription.py -k "test_sca_end_to_end or test_positive_candlepin_events_processed_by_stomp or test_positive_expired_SCA_cert_handling"
trigger: test-robottelo pytest: tests/foreman/api/test_subscription.py -k 'test_sca_end_to_end or test_positive_candlepin_events_processed_by_stomp or test_positive_expired_SCA_cert_handling'
trigger: test-robottelo pytest: tests/foreman/api/test_subscription.py -k 'test_sca_end_to_end or test_positive_candlepin_events_processed_by_stomp or test_positive_expired_SCA_cert_handling nailgun: 850
trigger: test-robottelo pytest: tests/foreman/api/test_subscription.py -k 'test_sca_end_to_end or test_positive_candlepin_events_processed_by_stomp or test_positive_expired_SCA_cert_handling nailgun: 850
trigger: test-robottelo pytest: tests/foreman/api/test_subscription.py -k 'test_sca_end_to_end or test_positive_candlepin_events_processed_by_stomp or test_positive_expired_SCA_cert_handling' nailgun: 850
test_subscription.py::test_positive_candlepin_events_processed_by_stomp is failing locally and in CI with the same error, which is unrelated to manifester.
I'm going to try one additional PRT run with all of the tests modified by this PR, excluding those that are failing in CI.
trigger: test-robottelo pytest: tests/foreman/api/ -k 'test_positive_add_rh or test_positive_add_rh_custom_spin or test_positive_update_rh_custom_spin or test_positive_publish_rh or test_positive_publish_rh_custom_spin or test_positive_promote_rh or test_positive_promote_rh_custom_spin or test_positive_sync_kickstart_repo or test_positive_sync_kickstart_check_os or test_positive_iso_library_sync or test_negative_check_org_endpoint or test_positive_filter_product_list or test_positive_generate_entitlements_report or test_positive_schedule_entitlements_report or test_positive_recreate_pulp_repositories or test_positive_sync_rh or test_positive_sync_repo_null_contents_changed or test_positive_reposet_enable or test_positive_reposet_disable or test_positive_synchronize_rh_product_future_sync_date or test_positive_synchronize_rh_product_past_sync_date or test_positive_subscription_status_disabled or test_positive_refresh or test_positive_create_after_refresh or test_negative_upload or test_positive_delete_manifest_as_another_user or test_positive_subscription_status_disabled or test_sca_end_to_end or test_positive_expired_SCA_cert_handling' nailgun: 850
The one failure in PRT build 1404 was the RHEL 6 run of test_subscription.py::test_positive_subscription_status_disabled, which is expected based on my comment above. I'm trying to determine which three content view tests were skipped and why.
Running the pytest command locally with the -rs --setup-plan options, it looks like the skipped tests were the Red Hat OSTree content view tests:
SKIPPED [1] tests/foreman/api/test_contentview.py:1402: BZ:1625783
SKIPPED [1] tests/foreman/api/test_contentview.py:1421: BZ:1625783
SKIPPED [1] tests/foreman/api/test_contentview.py:1439: BZ:1625783
I'm not quite sure what those tests matched on, since I did not intend to run them. But overall, I would say that the PRT run looks good. The only failure is expected and not caused by manifester.