Implement an automated test to cover Avocado features/options efficiently.

[willianrampazzo]

This issue tracks the implementation of an automated test to cover the Avocado features/options efficiently, as discussed here: #3969.

Features list:

🗷 assets.fetch.ignore_errors 🗷 assets.fetch.references 🗷 config.datadir ☐ core.input_encoding ☐ core.paginator ☐ core.show 🗷 core.verbose ☐ datadir.paths.base_dir ☐ datadir.paths.cache_dirs ☐ datadir.paths.data_dir ☐ datadir.paths.logs_dir ☐ datadir.paths.test_dir ☐ diff.create_reports ☐ diff.filter ☐ diff.html ☐ diff.jobids ☐ diff.open_browser ☐ diff.strip_id ☐ distro.distro_def_arch ☐ distro.distro_def_create ☐ distro.distro_def_name ☐ distro.distro_def_path ☐ distro.distro_def_release ☐ distro.distro_def_type ☐ distro.distro_def_version ☐ filter.by_tags.include_empty ☐ filter.by_tags.include_empty_key ☐ filter.by_tags.tags 🗹 job.output.loglevel ☐ job.replay.source_job_id 🗹 job.run.result.html.enabled ☐ job.run.result.html.open_browser 🗹 job.run.result.html.output 🗹 job.run.result.json.enabled 🗹 job.run.result.json.output 🗹 job.run.result.tap.enabled 🗹 job.run.result.tap.include_logs 🗹 job.run.result.tap.output 🗹 job.run.result.xunit.enabled 🗹 job.run.result.xunit.job_name 🗹 job.run.result.xunit.max_test_log_chars 🗹 job.run.result.xunit.output 🗹 job.run.timeout 🗷 jobs.get.output_files.destination 🗷 jobs.get.output_files.job_id 🗷 jobs.show.job_id ☐ json.variants.load ☐ list.external_runner ☐ list.external_runner_chdir ☐ list.external_runner_testdir ☐ list.loaders ☐ list.recipes.write_to_directory 🗷 list.references ☐ list.resolver ☐ nrunner.max_parallel_tasks ☐ nrunner.shuffle ☐ nrunner.spawner ☐ nrunner.status_server_uri ☐ plugins.cli.cmd.order ☐ plugins.cli.order 🗹 plugins.disable ☐ plugins.init.order ☐ plugins.job.prepost.order ☐ plugins.jobscripts.post ☐ plugins.jobscripts.pre ☐ plugins.jobscripts.warn_non_existing_dir ☐ plugins.jobscripts.warn_non_zero_status ☐ plugins.resolver.order ☐ plugins.result.order ☐ plugins.result_events.order ☐ plugins.result_upload.cmd ☐ plugins.result_upload.url ☐ plugins.resultsdb.api_url ☐ plugins.resultsdb.logs_url ☐ plugins.resultsdb.note_size_limit ☐ plugins.runnable.runner.order ☐ plugins.runner.order ☐ plugins.skip_broken_plugin_notification ☐ plugins.spawner.order ☐ plugins.varianter.order ☐ run.cit.combination_order ☐ run.cit.parameter_file 🗹 run.dry_run.enabled 🗹 run.dry_run.no_cleanup ☐ run.execution_order ☐ run.external_runner ☐ run.external_runner_chdir ☐ run.external_runner_testdir 🗹 run.failfast 🗹 run.ignore_missing_references 🗹 run.job_category 🗹 run.journal.enabled 🗹 run.keep_tmp ☐ run.loaders ☐ run.log_test_data_directories ☐ run.output_check ☐ run.output_check_record ☐ run.pict_binary ☐ run.pict_combinations_order ☐ run.pict_parameter_file ☐ run.pict_parameter_path ☐ run.references ☐ run.replay.ignore ☐ run.replay.job_id ☐ run.replay.resume ☐ run.replay.test_status 🗹 run.results.archive ☐ run.results_dir ☐ run.store_logging_stream ☐ run.test_parameters ☐ run.test_runner 🗹 run.unique_job_id ☐ run.wrapper.wrappers ☐ runner.output.color ☐ runner.output.colored ☐ runner.output.utf8 ☐ runner.timeout.after_interrupted ☐ runner.timeout.process_alive ☐ runner.timeout.process_died ☐ simpletests.status.skip_location ☐ simpletests.status.skip_regex ☐ simpletests.status.warn_location ☐ simpletests.status.warn_regex ☐ spawner.podman.bin ☐ spawner.podman.image ☐ sysinfo.collect.commands_timeout 🗹 sysinfo.collect.enabled ☐ sysinfo.collect.installed_packages ☐ sysinfo.collect.locale ☐ sysinfo.collect.optimize ☐ sysinfo.collect.per_test ☐ sysinfo.collect.profiler ☐ sysinfo.collect.sysinfodir ☐ sysinfo.collectibles.commands ☐ sysinfo.collectibles.fail_commands ☐ sysinfo.collectibles.fail_files ☐ sysinfo.collectibles.files ☐ sysinfo.collectibles.profilers ☐ variants.cit.combination_order ☐ variants.cit.parameter_file ☐ variants.contents ☐ variants.debug ☐ variants.inherit ☐ variants.json_variants_dump ☐ variants.pict_binary ☐ variants.pict_combinations_order ☐ variants.pict_parameter_file ☐ variants.pict_parameter_path ☐ variants.summary ☐ variants.tree ☐ variants.variants ☐ vmimage.get.arch ☐ vmimage.get.distro ☐ vmimage.get.version ☐ yaml_to_mux.files ☐ yaml_to_mux.filter_only ☐ yaml_to_mux.filter_out ☐ yaml_to_mux.inject ☐ yaml_to_mux.parameter_paths

🗹 - Already covered in the test. 🗷 - Plugin related, will not be tested with the Job API. ☐ - Still needs a test.

[willianrampazzo]

The base for the test was introduced at #4079.

[clebergnu]

Moving this to sprint 83.

[clebergnu]

While this is a worthy goal, the list of options is outdated given the age of this issue. I'm closing this, and we can consider opening a similar one in the future.