robottelo
robottelo copied to clipboard
SatelliteQE
Convert CLI activation key tests to manifester
This PR will convert all CLI tests that use manifests to obtain those manifests via manifester. Like https://github.com/SatelliteQE/robottelo/pull/10162, my plan is to push one commit for each module that I'm converting. The PR will remain in draft state until I have converted all affected modules. The list of CLI modules that I will be converting is as follows:
test_activationkey.py
test_contentaccess.py
test_contentview.py
test_errata.py
test_logging.py
test_remoteexecution.py
test_reporttemplate.py
test_repository_set.py
test_rhcloud_insights.py
test_rhcloud_inventory.py
test_satellitesync.py
test_subscription.py
test_syncplan.py
test_vm_install_products_package.py
trigger: test-robottelo pytest: tests/foreman/cli/test_activationkey.py -k "test_positive_create_with_name or test_positive_add_redhat_product or test_positive_add_redhat_and_custom_products or test_positive_delete_manifest or test_positive_delete_subscription or test_create_ak_with_syspurpose_set or test_update_ak_with_syspurpose_values or test_positive_add_subscription_by_id or test_positive_copy_subscription or test_positive_view_subscriptions_by_non_admin_user"
I'm holding off on running PRT for the content access tests, as two of them require https://github.com/SatelliteQE/manifester/pull/16 and dependabot has not yet run to update the manifester requirement to the latest release version.
Four of the CLI errata tests are failing both locally and in CI with causes unrelated to manifester:
test_host_errata_search_commands
test_apply_errata_using_default_content_view[rhel7_contenthost_module0]
test_update_applicable_package_using_default_content_view[rhel7_contenthost_module0]
test_install_applicable_package_to_registerd_host[rhel7_contenthost_module0]
PRT is showing 18 failures on cli/test_errata.py, which is inconsistent with what I'm seeing locally:
=================================================================================================================================== short test summary info ===================================================================================================================================
FAILED tests/foreman/cli/test_errata.py::test_host_errata_search_commands - AssertionError: assert 'dhcp-2-106.vms.sat.rdu2.redhat.com' in []
FAILED tests/foreman/cli/test_errata.py::test_apply_errata_using_default_content_view[rhel7_contenthost_module0] - robottelo.cli.base.CLIReturnCodeError: CLIReturnCodeError(status=70, stderr='Task 7f2c4d43-6485-44af-9746-24984a462c26 running: 0.17/1, 17%, elapsed: 00:00:00\nTask 7f2c...
FAILED tests/foreman/cli/test_errata.py::test_update_applicable_package_using_default_content_view[rhel7_contenthost_module0] - robottelo.cli.base.CLIReturnCodeError: CLIReturnCodeError(status=70, stderr='Task ac495a86-677b-488d-ba66-afd5179058cf running: 0.17/1, 17%, elapsed: 00:00:...
FAILED tests/foreman/cli/test_errata.py::test_install_applicable_package_to_registerd_host[rhel7_contenthost_module0] - AssertionError: assert 1 == 0
ERROR tests/foreman/cli/test_errata.py::test_errata_list_by_contentview_filter - AttributeError: 'Host' object has no attribute '_cont_inst_p'
========================================================================================================= 4 failed, 38 passed, 1 skipped, 127 warnings, 1 error in 2126.59s (0:35:26) ========================================================================================================
I'll try to determine the reason for the discrepancy.
The unexpected errors match https://bugzilla.redhat.com/show_bug.cgi?id=2042473, which was closed as not a bug. Trying another run in case this was a fluke of some kind.
trigger: test-robottelo pytest: tests/foreman/cli/test_reporttemplates.py -k "test_positive_generate_entitlements_report_multiple_formats or test_positive_schedule_entitlements_report or test_positive_generate_hostpkgcompare"
trigger: test-robottelo pytest: tests/foreman/cli/test_reporttemplates.py -k "test_positive_generate_entitlements_report_multiple_formats or test_positive_schedule_entitlements_report or test_positive_generate_hostpkgcompare"
test_positive_list_available_repositories is failing locally and in CI with the same error which is unrelated to manifester, so I'm expecting one failure from the above PRT run.
PRT is not currently working, but the rh_cloud tests are passing locally:
$ pytest tests/foreman/cli/test_rhcloud_insights.py
============================================================================================================================= test session starts ==============================================================================================================================
platform linux -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
shared_function enabled - OFF - scope: - storage: file
rootdir: /home/dsynk/qe_forks/robottelo, configfile: pyproject.toml
plugins: xdist-3.0.2, services-2.2.1, reportportal-5.1.2, mock-3.10.0, ibutsu-2.2.4, cov-3.0.0
collected 2 items
tests/foreman/cli/test_rhcloud_insights.py .. [100%]
================================================================================================================== 2 passed, 1 warning in 1841.93s (0:30:41) ===================================================================================================================
$ pytest tests/foreman/cli/test_rhcloud_inventory.py
============================================================================================================================= test session starts ==============================================================================================================================
platform linux -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
shared_function enabled - OFF - scope: - storage: file
rootdir: /home/dsynk/qe_forks/robottelo, configfile: pyproject.toml
plugins: xdist-3.0.2, services-2.2.1, reportportal-5.1.2, mock-3.10.0, ibutsu-2.2.4, cov-3.0.0
collected 7 items / 3 deselected / 4 selected
tests/foreman/cli/test_rhcloud_inventory.py .... [100%]
========================================================================================================== 4 passed, 3 deselected, 18 warnings in 1449.98s (0:24:09) ===========================================================================================================
trigger: test-robottelo pytest: tests/foreman/cli/test_satellitesync.py -k "test_positive_export_complete_version_rh_repo or test_positive_export_import_rh_cv or test_positive_export_import_redhat_cv_with_huge_contents or test_negative_import_redhat_cv_without_manifest or test_positive_complete_library_rh_repo"
trigger: test-robottelo pytest: tests/foreman/cli/test_satellitesync.py -k "test_positive_export_complete_version_rh_repo or test_positive_export_import_redhat_cv or test_positive_export_import_redhat_cv_with_huge_contents or test_negative_import_redhat_cv_without_manifest or test_positive_complete_library_rh_repo"
trigger: test-robottelo pytest: tests/foreman/cli/test_satellitesync.py -k 'test_positive_export_complete_version_rh_repo or test_positive_export_import_redhat_cv or test_positive_export_import_redhat_cv_with_huge_contents or test_negative_import_redhat_cv_without_manifest or test_positive_complete_library_rh_repo'
One of the failures from the last PRT run was a dictionary AttributeError that should now be resolved. the other two were due to manifester hitting its limit of 50 polling attempts while waiting for the manifest export job to complete on the RHSM side. I just pinged about the latter issue in the Teamnado channel, and the initial response I received was that 50 attempts (which translates to about 50 seconds if no other requests are hitting our RHSM organization) may not be enough time to guarantee that export jobs complete. Pending some additional feedback from Teamnado, I will likely increase the default number of attempts.
Teamnado reported that RHSM has been running a couple of large jobs for the last few days, so the failures that I saw were likely a result of resource competition. They have scaled up the resources, so I'll rerun the PRT job. Generally speaking, it looks like the default of 50 attempts is viable.
trigger: test-robottelo pytest: tests/foreman/cli/test_satellitesync.py -k 'test_positive_export_complete_version_rh_repo or test_positive_export_import_redhat_cv or test_positive_export_import_redhat_cv_with_huge_contents or test_negative_import_redhat_cv_without_manifest or test_positive_complete_library_rh_repo'
trigger: test-robottelo pytest: tests/foreman/cli -k 'test_negative_update_red_hat_repo or test_positive_create_with_name or test_positive_add_redhat_product or test_positive_add_redhat_and_custom_products or test_positive_delete_manifest or test_positive_delete_subscription or test_create_ak_with_syspurpose_set or test_update_ak_with_syspurpose_values or test_positive_add_subscription_by_id or test_positive_copy_subscription or test_positive_view_subscriptions_by_non_admin_user or test_positive_list_installable_updates or test_positive_erratum_installable or test_negative_rct_not_shows_golden_ticket_enabled or test_positive_rct_shows_golden_ticket_enabled or test_negative_unregister_and_pull_content or test_positive_add_rh_repo_by_id or test_positive_add_rh_repo_by_id_and_create_filter or test_positive_promote_rh_content or test_positive_promote_rh_and_custom_content or test_positive_publish_rh_content or test_positive_publish_rh_and_custom_content or test_positive_republish_after_rh_content_removed or test_positive_subscribe_chost_by_id_using_rh_content or test_positive_subscribe_chost_by_id_using_rh_content_and_filters or test_positive_install_by_host_collection_and_org or test_negative_install_by_hc_id_without_errata_info or test_negative_install_by_hc_name_without_errata_info or test_negative_install_without_hc_info or test_negative_install_by_hc_id_without_org_info or test_negative_install_by_hc_name_without_org_info or test_positive_list_affected_chosts or test_install_errata_to_one_host'
trigger: test-robottelo pytest: tests/foreman/cli -k 'test_positive_list_affected_chosts_by_erratum_restrict_flag or test_positive_list_filter_by_org_sort_by_date or test_positive_list_filter_by_cve or test_downgrade_applicable_package_using_default_content_view or test_downgrading_package_shows_errata_from_library or test_positive_logging_from_candlepin or test_positive_install_ansible_collection or test_positive_generate_entitlements_report_multiple_formats or test_positive_schedule_entitlements_report or test_positive_generate_hostpkgcompare or test_positive_enable_by_name or test_positive_enable_by_label or test_positive_enable_by_id or test_positive_disable_by_name or test_positive_disable_by_label or test_positive_connection_option or test_positive_inventory_generate_upload_cli or test_positive_inventory_recommendation_sync or test_positive_sync_inventory_status or test_positive_export_complete_version_rh_repo or test_positive_complete_library_rh_repo or test_positive_export_import_redhat_cv or test_positive_export_import_redhat_cv_with_huge_contents or test_negative_import_redhat_cv_without_manifest or test_positive_manifest_upload or test_positive_manifest_delete or test_positive_enable_manifest_reposet or test_positive_manifest_history or test_positive_manifest_refresh or test_positive_subscription_list or test_positive_delete_manifest_as_another_user or test_positive_auto_attach_disabled_golden_ticket or test_positive_synchronize_rh_product_past_sync_date or test_positive_synchronize_rh_product_future_sync_date or test_positive_synchronize_rh_product_future_sync_date'
Posting PRT results here for visibility. This is from the first of the two runs above:
11:44:54 ============================= test session starts ==============================
11:44:54 platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0
11:44:54 shared_function enabled - OFF - scope: - storage: file
11:44:54 rootdir: /opt/app-root/src/robottelo, configfile: pyproject.toml
11:44:54 plugins: ibutsu-2.2.4, mock-3.10.0, reportportal-5.1.3, services-2.2.1, xdist-3.0.2
11:45:12 collected 1869 items / 1729 deselected / 140 selected
11:45:12
12:10:13 tests/foreman/cli/test_activationkey.py ................ [ 11%]
12:10:44 tests/foreman/cli/test_capsule.py ......F [ 16%]
12:10:47 tests/foreman/cli/test_computeresource_libvirt.py . [ 17%]
12:12:35 tests/foreman/cli/test_contentaccess.py ..... [ 20%]
12:18:18 tests/foreman/cli/test_contentview.py ............F........... [ 37%]
12:21:50 tests/foreman/cli/test_contentviewfilter.py ............................ [ 57%]
12:21:50 [ 57%]
12:22:21 tests/foreman/cli/test_discoveryrule.py ....... [ 62%]
12:22:51 tests/foreman/cli/test_docker.py ....... [ 67%]
12:39:01 tests/foreman/cli/test_errata.py .............. [ 77%]
12:39:07 tests/foreman/cli/test_operatingsystem.py ....... [ 82%]
12:40:06 tests/foreman/cli/test_repository.py ..............s [ 93%]
12:41:05 tests/foreman/cli/test_subscription.py . [ 94%]
12:41:41 tests/foreman/cli/test_syncplan.py ....... [ 99%]
12:43:36 tests/foreman/cli/test_template.py . [100%]
It looks like my -k argument matched on some tests that were not actually modified by this PR. This includes the two failures, which were test_capsule.py::test_positive_create_with_name and test_contentview.py::TestContentView::test_positive_create_with_name.
Results from the second PRT run:
11:55:04 ============================= test session starts ==============================
11:55:04 platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0
11:55:04 shared_function enabled - OFF - scope: - storage: file
11:55:04 rootdir: /opt/app-root/src/robottelo, configfile: pyproject.toml
11:55:04 plugins: ibutsu-2.2.4, mock-3.10.0, reportportal-5.1.3, services-2.2.1, xdist-3.0.2
11:55:26 collected 1869 items / 1827 deselected / 42 selected
11:55:26
12:28:13 tests/foreman/cli/test_errata.py ..........EE [ 28%]
12:28:35 tests/foreman/cli/test_logging.py . [ 30%]
12:37:32 tests/foreman/cli/test_remoteexecution.py F [ 33%]
12:49:25 tests/foreman/cli/test_reporttemplates.py F.. [ 40%]
12:52:31 tests/foreman/cli/test_repository_set.py ..... [ 52%]
13:29:51 tests/foreman/cli/test_rhcloud_insights.py .. [ 57%]
13:52:51 tests/foreman/cli/test_rhcloud_inventory.py ... [ 64%]
15:20:57 tests/foreman/cli/test_satellitesync.py ..... [ 76%]
15:25:55 tests/foreman/cli/test_subscription.py ....F... [ 95%]
15:41:14 tests/foreman/cli/test_syncplan.py .F [100%]
It does not look like there were any extra tests matched by this set of -k arguments, so I will investigate these failures.