Pytest adoption tracker

[trexfeathers]

Iris approximate pytest adoption

Generated using this script

Click to expand this section...

import ast
from pathlib import Path
from sys import argv

# Also requires the `tabulate` package.
import pandas as pd


if not (len(argv) == 2 and Path(argv[1]).exists()):
    message = "Correct script usage = `pytest_adoption.py <iris_root_dir>`"
    raise ValueError(message)

IRIS_ROOT = Path(argv[1])
IRIS_TREE_URL = "https://github.com/SciTools/iris/tree/main/"
markdown_path = Path(__file__).with_suffix(".md")

tests_dir = IRIS_ROOT / "lib" / "iris" / "tests"
dataframe_data = []
total_asserts = 0
total_asserts_pytest = 0

for file_path in tests_dir.rglob("*.py"):
    file_text = file_path.read_text()
    parsed = ast.parse(source=file_text)
    calls = filter(lambda node: hasattr(node, "func"), ast.walk(parsed))
    assert_calls = filter(
        lambda c: getattr(c.func, "attr", "")[:6] == "assert", calls
    )
    assert_calls_methods = filter(lambda c: hasattr(c.func, "value"), assert_calls)
    assert_calls_methods_self = filter(
        lambda m: getattr(m.func.value, "id", "") == "self", assert_calls_methods
    )
    num_asserts_unittest = len(list(assert_calls_methods_self))

    assert_keywords = filter(
        lambda node: isinstance(node, ast.Assert), ast.walk(parsed)
    )
    num_asserts_pytest = len(list(assert_keywords))

    both_asserts = num_asserts_unittest + num_asserts_pytest
    if both_asserts == 0:
        percent_adoption = "N/A"
    else:
        percent_adoption = f"{num_asserts_pytest / both_asserts * 100:.0f}"

    total_asserts += both_asserts
    total_asserts_pytest += num_asserts_pytest

    file_url = IRIS_TREE_URL + str(file_path.relative_to(IRIS_ROOT))
    file_link_text = f"{file_path.relative_to(tests_dir).stem}"

    dataframe_data.append(
        {
            "% adoption": percent_adoption,
            # "File Path": f"[{file_link_text}]({file_url})",
            "File Path": f"{file_link_text}",
            "Number of `self.assertSomething()`": num_asserts_unittest,
            "Number of `assert something`": num_asserts_pytest,
        }
    )

pytest_adoption_df = pd.DataFrame(dataframe_data)

total_percent_adoption = f"{total_asserts_pytest / total_asserts * 100:.0f}%"
total_line = (
    "## Total adoption\n\n"
    f"### {total_asserts_pytest} out of {total_asserts} : {total_percent_adoption}"
)

script_line = (
    "## Generated using this script\n\n"
    "<details>\n"
    "<summary>Click to expand <b>this section...</b></summary>\n\n"
    "```python\n"
    f"{Path(__file__).read_text()}"
    "```\n\n"
    "</details>"
)

# pytest_adoption_df.to_markdown(markdown_path, index=False)
pytest_adoption_df.to_html(markdown_path, index=False)

full_text = (
    "# Iris approximate pytest adoption\n\n"
    f"{script_line}\n\n"
    f"{total_line}\n\n"
    "## Breakdown\n\n"
    f"{markdown_path.read_text()}"
)

markdown_path.write_text(full_text)

Total adoption

548 out of 10272 : 5%

Breakdown

% adoption	File Path	Number of `self.assertSomething()`	Number of `assert something`
0	test_nimrod	6	0
0	test_cube_to_pp	15	0
0	test_basic_maths	152	0
0	test_coord_api	138	0
N/A	test_imports	0	0
0	test_file_load	5	0
0	test_util	29	0
0	test_cube	9	0
0	test_file_save	15	0
0	test_hybrid	45	0
0	test_image_json	4	0
0	test_pp_to_cube	15	0
100	test_coding_standards	0	4
0	test_cell	90	0
0	test_peak	29	0
0	test_intersect	2	0
0	test_io_init	6	0
0	test_iterate	74	0
0	test_mapping	3	0
0	test_pp_module	88	0
0	system_test	1	0
0	test_aggregate_by	62	0
65	test_lazy_aggregate_by	6	11
0	test_pp_stash	53	0
2	test_merge	100	2
0	test_pickling	15	0
0	test_coordsystem	90	0
0	test_constraints	119	0
0	test_uri_callback	1	0
20	test_analysis	166	41
0	test_cf	92	0
0	test_concatenate	161	0
0	test_cartography	3	0
0	test_abf	3	0
0	test_plot	24	0
0	test_name	9	0
0	__init__	20	0
0	test_cdm	265	0
0	test_ff	42	0
0	test_load	21	0
0	test_netcdf	102	0
0	test_analysis_calculus	62	0
0	test_pp_cf	3	0
0	test_quickplot	7	0
0	test_std_names	5	0
0	test_raster	3	0
N/A	__init__	0	0
N/A	__init__	0	0
2	test_regrid_area_weighted_rectilinear_src_and_grid	54	1
0	test_regrid_conservative_via_esmpy	40	0
N/A	__init__	0	0
N/A	idiff	0	0
N/A	recreate_imagerepo	0	0
0	test_pp	74	0
0	test_regrid_equivalence	8	0
0	test_pp_constrained_load_cubes	3	0
0	test_Datums	1	0
N/A	__init__	0	0
0	test_PartialDateTime	1	0
0	test_ff	11	0
0	test_new_axis	3	0
0	test_subset	1	0
0	test_cube	5	0
100	test_netcdf__loadsaveattrs	0	23
0	test_climatology	3	0
0	test_pickle	1	0
0	test_trajectory	13	0
0	test_regridding	11	0
N/A	__init__	0	0
0	test_area_weighted	4	0
N/A	__init__	0	0
0	test_OceanSigmaZFactory	6	0
N/A	__init__	0	0
14	test_concatenate	31	5
N/A	__init__	0	0
0	test_CubeRepresentation	33	0
0	test_regrid_ProjectedUnstructured	13	0
0	test_ugrid_load	9	0
0	test_ugrid_save	5	0
N/A	__init__	0	0
3	test_fast_load	28	1
N/A	__init__	0	0
0	test_merge	3	0
100	test_thread_safety	0	7
100	test_delayed_save	0	36
0	test_attributes	5	0
12	test_aux_factories	7	1
5	test_general	39	2
N/A	__init__	0	0
100	test_self_referencing	0	2
100	test__dask_locks	0	11
0	test_coord_systems	11	0
N/A	__init__	0	0
0	test_colorbar	6	0
N/A	test_animate	0	0
0	test_netcdftime	1	0
0	test_nzdateline	1	0
N/A	test_plot_2d_coords	0	0
0	test_vector_plots	1	0
N/A	__init__	0	0
0	test_fieldsfile	2	0
100	_stock_2d_latlons	0	2
N/A	__init__	0	0
N/A	mesh	0	0
100	netcdf	0	1
N/A	__init__	0	0
N/A	conftest	0	0
0	test_Future	14	0
0	test_sample_data_path	8	0
0	test_VARIANCE	12	0
0	test_COUNT	14	0
0	test_Nearest	5	0
0	test_PROPORTION	7	0
0	test_PointInCell	1	0
0	test_STD_DEV	9	0
0	test_WPERCENTILE	40	0
0	test__axis_to_single_trailing	8	0
0	test_PercentileAggregator	19	0
N/A	__init__	0	0
0	test_Aggregator	21	0
0	test_MAX	9	0
0	test_MEAN	11	0
0	test_MIN	9	0
0	test_AreaWeighted	4	0
0	test_Linear	5	0
0	test_RMS	18	0
0	test_SUM	27	0
0	test_PERCENTILE	32	0
0	test_MAX_RUN	6	0
0	test_WeightedPercentileAggregator	21	0
N/A	__init__	0	0
0	test_AreaWeightedRegridder	20	0
0	test_gridcell_angles	31	0
0	test_project	15	0
0	test_rotate_grid_vectors	9	0
2	test_rotate_winds	65	1
100	test__get_lon_lat_coords	0	6
0	test__quadrant_area	9	0
60	test__xy_range	2	3
N/A	__init__	0	0
0	test_area_weights	2	0
N/A	__init__	0	0
0	test_geometry_area_weights	10	0
0	test__extract_relevant_cube_slice	5	0
N/A	__init__	0	0
0	test_RectilinearInterpolator	65	0
0	test_get_xy_dim_coords	15	0
N/A	test__arith__dask_array	0	0
0	test_divide	4	0
0	__init__	28	0
0	test_add	2	0
0	test_multiply	2	0
0	test_subtract	2	0
N/A	test__arith__derived_coords	0	0
6	test__arith__meshcoords	16	1
0	test__get_dtype	1	0
0	test__inplace_common_checks	21	0
0	test__output_dtype	8	0
N/A	__init__	0	0
1	test_RectilinearRegridder	146	2
40	test__CurvilinearRegridder	12	8
N/A	__init__	0	0
0	test__RegularGridInterpolator	8	0
N/A	__init__	0	0
0	test_pearsonr	15	0
N/A	__init__	0	0
0	test_UnstructuredNearestNeighbourRegridder	19	0
0	test__nearest_neighbour_indices_ndcoords	12	0
0	test_Trajectory	35	0
53	test_interpolate	14	16
31	test_AtmosphereSigmaFactory	18	8
0	test_OceanSFactory	32	0
0	test_OceanSg1Factory	31	0
0	test_OceanSigmaFactory	18	0
0	test_AuxCoordFactory	26	0
N/A	__init__	0	0
0	test_HybridPressureFactory	32	0
0	test_OceanSg2Factory	31	0
0	test_OceanSigmaZFactory	43	0
N/A	__init__	0	0
N/A	__init__	0	0
0	test_Lenient	49	0
0	test__Lenient	120	0
0	test__lenient_client	36	0
0	test__lenient_service	20	0
0	test__qualname	5	0
0	test_BaseMetadata	284	0
0	test_CoordMetadata	113	0
0	test__NamedTupleMeta	23	0
0	test_metadata_filter	23	0
N/A	__init__	0	0
0	test_AncillaryVariableMetadata	89	0
0	test_CellMeasureMetadata	113	0
54	test_CubeMetadata	50	59
0	test_hexdigest	26	0
0	test_metadata_manager_factory	23	0
N/A	__init__	0	0
0	test_CFVariableMixin	50	0
0	test_LimitedAttributeDict	9	0
0	test__get_valid_standard_name	11	0
N/A	__init__	0	0
0	test_Resolve	546	0
100	__init__	0	2
100	test__CoordMetaData	0	5
100	test__CoordSignature	0	4
0	test__CubeSignature	10	0
0	test_concatenate	49	0
N/A	__init__	0	0
0	test_NetCDF	6	0
N/A	__init__	0	0
0	test_Constraint_equality	49	0
0	test_NameConstraint	48	0
N/A	__init__	0	0
0	test_add_categorised_coord	2	0
0	test_add_hour	4	0
100	test_coord_categorisation	0	2
0	test_Stereographic	16	0
0	test_TransverseMercator	6	0
0	test_Mercator	18	0
0	test_PolarStereographic	23	0
0	test_LambertAzimuthalEqualArea	10	0
N/A	__init__	0	0
0	test_GeogCS	5	0
0	test_LambertConformal	16	0
0	test_Orthographic	6	0
N/A	test_RotatedMercator	0	0
0	test_RotatedPole	8	0
0	test_Geostationary	8	0
100	test_ObliqueMercator	0	1
0	test_VerticalPerspective	6	0
0	test_AlbersEqualArea	18	0
0	test_DimCoord	85	0
0	test__DimensionalMetadata	51	0
0	test_CellMethod	6	0
0	test_AncillaryVariable	70	0
0	test_CellMeasure	18	0
2	test_Coord	146	3
0	__init__	15	0
0	test_AuxCoord	118	0
0	test_Cell	39	0
N/A	__init__	0	0
0	test_Cube__operators	10	0
100	test_CubeAttrsDict	0	45
3	test_CubeList	64	2
4	test_Cube__aggregated_by	112	5
5	test_Cube	497	27
N/A	__init__	0	0
0	test_DataManager	141	0
N/A	__init__	0	0
N/A	__init__	0	0
0	test_export_geotiff	7	0
N/A	__init__	0	0
0	test_regrid_area_weighted_rectilinear_src_and_grid	11	0
0	test_regrid_weighted_curvilinear_to_rectilinear	18	0
N/A	__init__	0	0
0	test_CubeListRepresentation	5	0
0	test_CubeRepresentation	57	0
N/A	__init__	0	0
0	test_relevel	6	0
N/A	__init__	0	0
N/A	__init__	0	0
20	test_CFUGridAuxiliaryCoordinateVariable	8	2
20	test_CFUGridConnectivityVariable	8	2
0	test_CFUGridGroup	5	0
17	test_CFUGridMeshVariable	10	2
0	test_CFUGridReader	5	0
N/A	__init__	0	0
0	test_ParseUgridOnLoad	8	0
0	test_load_mesh	3	0
0	test_load_meshes	20	0
N/A	__init__	0	0
0	test_Connectivity	43	0
0	test_Mesh	147	0
2	test_MeshCoord	100	2
0	test_Mesh__from_coords	41	0
N/A	__init__	0	0
0	test_ConnectivityMetadata	121	0
0	test_MeshCoordMetadata	113	0
0	test_MeshMetadata	121	0
N/A	__init__	0	0
0	test_recombine_submeshes	49	0
0	__init__	6	0
7	test_rules	28	2
N/A	__init__	0	0
0	test_ABFField	2	0
N/A	__init__	0	0
0	test_CFGroup	1	0
0	test_CFReader	48	0
N/A	__init__	0	0
50	test__dot_path	4	4
N/A	__init__	0	0
0	test_ArakawaC	6	0
0	test_Grid	10	0
0	test_NewDynamics	3	0
0	test_ENDGame	3	0
0	test_FFHeader	7	0
0	test_FF2PP	32	0
N/A	__init__	0	0
0	test__cf_height_from_name	23	0
0	test__build_cell_methods	7	0
0	test__build_lat_lon_for_NAME_timeseries	18	0
0	test__calc_integration_period	7	0
0	test__generate_cubes	22	0
N/A	__init__	0	0
0	test__grid_mappings	22	0
0	test__hybrid_formulae	3	0
27	test__latlon_dimcoords	8	3
0	test__miscellaneous	20	0
20	test__time_coords	16	4
0	__init__	2	0
N/A	__init__	0	0
0	test_engine	17	0
0	test_build_cube_metadata	6	0
43	test_build_dimension_coordinate	8	6
0	test_build_geostationary_coordinate_system	1	0
0	test_build_lambert_conformal_coordinate_system	1	0
0	test_build_stereographic_coordinate_system	1	0
0	test_build_transverse_mercator_coordinate_system	1	0
0	test_build_verticalp_coordinate_system	1	0
0	test_get_attr_units	2	0
0	test_get_cf_bounds_var	2	0
0	test_has_supported_mercator_parameters	7	0
0	test_reorder_bounds_data	3	0
0	test_build_polar_stereographic_coordinate_system	5	0
0	test_get_names	4	0
0	test_has_supported_polar_stereographic_parameters	15	0
100	test_build_ancil_var	0	1
100	test_build_cell_measure	0	1
N/A	test_build_oblique_mercator_coordinate_system	0	0
0	test_parse_cell_methods	12	0
N/A	__init__	0	0
0	test_build_albers_equal_area_coordinate_system	1	0
20	test_build_auxiliary_coordinate	4	1
0	test_build_lambert_azimuthal_equal_area_coordinate_system	1	0
0	test_build_mercator_coordinate_system	5	0
N/A	__init__	0	0
N/A	__init__	0	0
0	test__load_cube	12	0
0	test__get_cf_var_data	9	0
0	test__load_aux_factory	40	0
0	test__translate_constraints_to_var_callback	10	0
0	test_load_cubes	28	0
100	test__chunk_control	0	37
0	test__data_fillvalue_check	11	0
100	test__fillvalue_report	0	5
0	test_Saver__ugrid	121	0
N/A	__init__	0	0
0	test_Saver__lazy	4	0
0	test_Saver	67	0
100	test_Saver__lazy_stream_data	0	12
16	test_save	21	4
N/A	__init__	0	0
0	test_vertical_coord	2	0
0	test_units	36	0
N/A	__init__	0	0
0	test_PPDataProxy	9	0
0	test__create_field_data	3	0
0	test__field_gen	7	0
0	test_as_fields	2	0
N/A	test_load	0	0
0	test_save_fields	4	0
0	test_save_pairs_from_cube	6	0
0	test_PPField	58	0
0	test__convert_constraints	13	0
0	test__interpret_field	14	0
16	test_save	31	6
12	test__data_bytes_to_shaped_array	7	1
0	test__convert_scalar_pseudo_level_coords	2	0
0	test__convert_time_coords	19	0
0	test__convert_vertical_coords	18	0
0	test__reshape_vector_args	6	0
0	test__model_level_number	2	0
N/A	__init__	0	0
0	test__dim_or_aux	2	0
0	test_convert	13	0
0	test__collapse_degenerate_points_and_bounds	16	0
0	test__convert_scalar_realization_coords	2	0
0	test__epoch_date_hours	14	0
0	test__reduced_points_and_bounds	23	0
0	test__all_other_rules	17	0
N/A	__init__	0	0
0	test__make_cube	3	0
0	test_Loader	8	0
N/A	__init__	0	0
3	test_ArrayStructure	31	1
0	test_GroupStructure	18	0
N/A	__init__	0	0
0	test_um_to_pp	3	0
0	test__convert_collation	25	0
0	test_FieldCollation	7	0
N/A	__init__	0	0
N/A	__init__	0	0
0	test_BasicFieldCollation	28	0
10	test_group_structured_fields	9	1
N/A	__init__	0	0
0	test_optimal_array_structure	32	0
N/A	__init__	0	0
0	test_expand_filespecs	11	0
0	test_run_callback	8	0
N/A	test__generate_cubes	0	0
N/A	test_save	0	0
0	test_co_realise_cubes	7	0
0	test_is_lazy_data	2	0
100	test_is_lazy_masked_data	0	1
0	test_lazy_elementwise	8	0
0	test_multidim_lazy_stack	3	0
0	test_non_lazy	4	0
N/A	__init__	0	0
0	test_as_concrete_data	17	0
0	test_as_lazy_data	13	0
0	test_map_complete_blocks	12	0
N/A	__init__	0	0
0	test_ProtoCube	9	0
N/A	__init__	0	0
37	test_pandas	80	46
0	test__fixup_dates	7	0
0	test__get_plot_objects	3	0
0	test_contour	5	0
0	test_contourf	5	0
100	test_hist	0	1
0	test_outline	5	0
N/A	test_pcolor	0	0
N/A	test_pcolormesh	0	0
0	test_points	5	0
0	_blockplot_common	7	0
0	test__get_plot_defn	3	0
0	test__replace_axes_with_cartopy_axes	2	0
0	test_scatter	4	0
0	__init__	6	0
0	test__check_bounds_contiguity_and_mask	2	0
0	test__check_geostationary_coords_and_convert	1	0
0	test__get_plot_defn_custom_coords_picked	13	0
0	test_plot	8	0
N/A	__init__	0	0
0	test_contour	2	0
0	test_contourf	2	0
0	test_outline	2	0
0	test_plot	6	0
0	test_points	2	0
0	test_scatter	2	0
0	test_pcolor	2	0
0	test_pcolormesh	2	0
N/A	__init__	0	0
N/A	__init__	0	0
0	test_CubePrintout	37	0
0	test_Table	34	0
N/A	__init__	0	0
2	test_CubeSummary	63	1
N/A	__init__	0	0
0	test_IrisTest	14	0
N/A	__init__	0	0
0	test_netcdf	9	0
N/A	__init__	0	0
0	test_PartialDateTime	16	0
0	test_demote_dim_coord_to_aux_coord	7	0
0	test_describe_diff	3	0
100	test_guess_coord_axis	0	3
0	test_squeeze	5	0
100	test_new_axis	0	26
0	test_reverse	38	0
0	test_rolling_window	9	0
0	test__slice_data_with_keys	10	0
100	test__mask_array	0	11
0	test_mask_cube	17	0
0	test_file_is_newer_than	8	0
N/A	__init__	0	0
0	test__is_circular	2	0
0	test_array_equal	29	0
0	test_column_slices_generator	2	0
0	test_broadcast_to_shape	7	0
0	test_promote_aux_coord_to_dim_coord	11	0
60	test_unify_time_units	2	3
0	test__coord_regular	17	0
0	test_find_discontiguities	7	0
64	test_equalise_attributes	5	9

[trexfeathers]

Sorry for the janky formatting - reached the character limit. But it's here as a proof of concept that could be automated.

Ping @ESadek-MO

[pp-mo]

Hi @trexfeathers Nice work, but a couple of things bothering me :

Firstly, it would be much handier to have a full path to each test module in the table Was this due to the line-length constraint you mention ?

Secondly, I think the entry for test_netcdf__loadsaveattrs is showing some problems : It is saying

%pytest	name	N-unittest	N-pytest
15	test_netcdf__loadsaveattrs	127	23

But actually, this module is entirely written in pytest. So something odd here ?

[trexfeathers]

@pp-mo

Firstly, it would be much handier to have a full path to each test module in the table Was this due to the line-length constraint you mention ?

100% - I did originally have the full path and it was even a link.

Secondly, I think the entry for test_netcdf__loadsaveattrs is showing some problems : It is saying

%pytest name N-unittest N-pytest 15 test_netcdf__loadsaveattrs 127 23 But actually, this module is entirely written in pytest. So something odd here ?

Good spot! That was copy-pasta. I have updated

-assert_calls_methods = filter(lambda c: hasattr(c.func, "value"), calls)
+assert_calls_methods = filter(lambda c: hasattr(c.func, "value"), assert_calls)

[rcomer]

pytestify is excellent, so if you can get a given test module to a point where it is using unittest.TestCase rather than a bespoke Iris test class, the rest should be is easy.

Apologies if I'm teaching Granny to suck eggs.

[trexfeathers]

if you can get a given test module to a point where it is using unittest.TestCase rather than a bespoke Iris test class, the rest should be is easy

I may have misunderstood. If I've understood correctly, then I don't think this is realistic. The use of IrisTest is super embedded throughout our testing. Things like get_data_path(), assertCML() and assertArrayEqual() are everywhere. Rewriting the tests to a point where an auto-converter can handle them would likely take a similar amount of time to just converting to PyTest ourselves.

Bonus points: if we manage to agree on some common conventions (e.g. how we use fixtures, conftest.py etcetera) BEFORE converting anything, then a tracker such as the above could also double up as a tracker of where we have rolled out these conventions (with a 5% error for the PyTest usages we have already written!)

[stephenworsley]

Translation Guide

Some methods inherited from unittest will need replacing with pytest friendly equivalents. For developer convenience while translating, here is a table of unittest methods and their replacement code. Feel free to add to or edit this table as appropriate.

unittest method	pytest equivalent
assertTrue(x)	assert x
assertFalse(x)	assert not x
assertRegex(x, y)	assert re.match(y, x)
assertRaisesRegex(cls, msg_re)	with pytest.raises(cls, match=msg_re)

[pp-mo]

Random Notes

1. best not to use @staticmethod on a fixture
   • this it's OK when it passes,
   • but a test-fail gives bad result like :"E AttributeError: 'staticmethod' object has no attribute '__name__'"