pytest-dev
Failure output order of dictionary keys is alphabetical instead of insertion order
- Create a file like this:
class TestDictOrder:
def test_dict_order(self):
a = {
"Existing Hash": "",
"Existing Modified": "",
"Existing Size": "",
"Existing Broken": "",
"Set Broken": "N",
"Head Status": "",
"Head Error": "",
"Get Status": "",
"Get Error": "",
"New ETag": "",
"ETag Changed": "",
"New Modified": "",
"Modified Changed": "",
"Modified Newer": "",
"Modified Value": "",
"New Size": "",
"Size Changed": "",
"New Hash": "",
"Hash Changed": "",
"Update": "N",
}
print(a)
assert a == {}
- run pytest with -vv
- The keys in the output failure message are not in insertion order (the guaranteed order since Python 3.7) but are displayed alphabetically instead (presumably because that's easier to implement?). I think there should be an option to have the output order be insertion order (or it should be the standard output order) as that can be helpful with debugging large and nested dictionaries and is the order when you print the dictionary.
In the output from pytest below, "Left contains 20 more items:" and "Full diff:" are alphabetical instead of insertion order:
1 > pytest -vv test_dict_order.py
=================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.13.3, pytest-8.4.0, pluggy-1.6.0 -- /home/mcarans/Code/VirtualEnvs/scratch/bin/python
cachedir: .pytest_cache
rootdir: /home/mcarans/Code/scratch
plugins: typeguard-4.4.2
collected 1 item
test_dict_order.py::TestDictOrder::test_dict_order FAILED [100%]
======================================================================================================================================== FAILURES =========================================================================================================================================
______________________________________________________________________________________________________________________________ TestDictOrder.test_dict_order ______________________________________________________________________________________________________________________________
self = <test_dict_order.TestDictOrder object at 0x7adebb01f890>
def test_dict_order(self):
a = {
"Existing Hash": "",
"Existing Modified": "",
"Existing Size": "",
"Existing Broken": "",
"Set Broken": "N",
"Head Status": "",
"Head Error": "",
"Get Status": "",
"Get Error": "",
"New ETag": "",
"ETag Changed": "",
"New Modified": "",
"Modified Changed": "",
"Modified Newer": "",
"Modified Value": "",
"New Size": "",
"Size Changed": "",
"New Hash": "",
"Hash Changed": "",
"Update": "N",
}
print(a)
> assert a == {}
E AssertionError: assert {'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'} == {}
E
E Left contains 20 more items:
E {'ETag Changed': '',
E 'Existing Broken': '',
E 'Existing Hash': '',
E 'Existing Modified': '',
E 'Existing Size': '',
E 'Get Error': '',
E 'Get Status': '',
E 'Hash Changed': '',
E 'Head Error': '',
E 'Head Status': '',
E 'Modified Changed': '',
E 'Modified Newer': '',
E 'Modified Value': '',
E 'New ETag': '',
E 'New Hash': '',
E 'New Modified': '',
E 'New Size': '',
E 'Set Broken': 'N',
E 'Size Changed': '',
E 'Update': 'N'}
E
E Full diff:
E - {}
E + {
E + 'ETag Changed': '',
E + 'Existing Broken': '',
E + 'Existing Hash': '',
E + 'Existing Modified': '',
E + 'Existing Size': '',
E + 'Get Error': '',
E + 'Get Status': '',
E + 'Hash Changed': '',
E + 'Head Error': '',
E + 'Head Status': '',
E + 'Modified Changed': '',
E + 'Modified Newer': '',
E + 'Modified Value': '',
E + 'New ETag': '',
E + 'New Hash': '',
E + 'New Modified': '',
E + 'New Size': '',
E + 'Set Broken': 'N',
E + 'Size Changed': '',
E + 'Update': 'N',
E + }
test_dict_order.py:26: AssertionError
---------------------------------------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------------------------------------
{'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'}
================================================================================================================================= short test summary info =================================================================================================================================
FAILED test_dict_order.py::TestDictOrder::test_dict_order - AssertionError: assert {'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'} == {}
Left contains 20 more items:
{'ETag Changed': '',
'Existing Broken': '',
'Existing Hash': '',
'Existing Modified': '',
'Existing Size': '',
'Get Error': '',
'Get Status': '',
'Hash Changed': '',
'Head Error': '',
'Head Status': '',
'Modified Changed': '',
'Modified Newer': '',
'Modified Value': '',
'New ETag': '',
'New Hash': '',
'New Modified': '',
'New Size': '',
'Set Broken': 'N',
'Size Changed': '',
'Update': 'N'}
Full diff:
- {}
+ {
+ 'ETag Changed': '',
+ 'Existing Broken': '',
+ 'Existing Hash': '',
+ 'Existing Modified': '',
+ 'Existing Size': '',
+ 'Get Error': '',
+ 'Get Status': '',
+ 'Hash Changed': '',
+ 'Head Error': '',
+ 'Head Status': '',
+ 'Modified Changed': '',
+ 'Modified Newer': '',
+ 'Modified Value': '',
+ 'New ETag': '',
+ 'New Hash': '',
+ 'New Modified': '',
+ 'New Size': '',
+ 'Set Broken': 'N',
+ 'Size Changed': '',
+ 'Update': 'N',
+ }
==================================================================================================================================== 1 failed in 0.03s ====================================================================================================================================
[
Pytest shows a order invariant difference between 2 dicts as dict equality doesn't require the order of items in a dictionary to match
Its not clear what you sre asking for
@RonnyPfannschmidt Sorry for the lack of clarity. I'm thinking only of the way the differences are displayed rather than changing dict equality.
assert {"c": 3, "d": 4, "b": 2, "a": 1} == {"d": 4, "c": 3}
Currently the output is:
Common items:
{'c': 3, 'd': 4}
Left contains 2 more items:
{'a': 1, 'b': 2}
Full diff:
{
+ 'a': 1,
+ 'b': 2,
'c': 3,
'd': 4,
}
I'm suggesting the output would follow the insertion order as far as possible and default to using the order of one of the sides for common items (I have chosen right hand side below) ie.:
Common items:
{'d': 4, 'c': 3}
Left contains 2 more items:
{'b': 2, 'a': 1}
Full diff:
{
'd': 4,
'c': 3,
+ 'b': 2,
+ 'a': 1,
}
So the request is for the display order to orient on the insert order
Based on the code it might be a mistake that the full diff is printed
In any case we need a deep diff to correctly implement this .
Currently pformat is used to get diff compatible left/right
Unless a correct deep diff we can apply here is avaliable im against implementing this in pytest
@RonnyPfannschmidt pformat has an option sort_dicts which defaults to True. It is described as "If True, dictionaries will be formatted with their keys sorted, otherwise they will be displayed in insertion order"
Is that what is sorting the keys alphabetically in the pytest output?
If not, I found a deep diff library called deepdiff. For the example above, it gives:
pprint(DeepDiff(a,b))
{'dictionary_item_removed': ["root['b']", "root['a']"]}
Is that what is sorting the keys alphabetically in the pytest output?
pformat is used here:
https://github.com/pytest-dev/pytest/blob/9e9633de9da7a9fab03b4bba3a326bf85b412050/src/_pytest/assertion/util.py#L498-L541
Looks like it would be just a matter of passing sort_dicts=False to those pformat calls.
We will have to ensure compatible dict ordering in nested structures to ensure minimal diffs
Order matching dicts in modern python make this easier but its still a caveat
This issue is stale because it has the status: needs information label and requested follow-up information was not provided for 14 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi, I’d like to work on this issue as my first contribution to the pytest repo.
Hi @RonnyPfannschmidt ! I’m interested in working on this issue as my first open-source contribution. Could you please assign it to me?
Hey @RonnyPfannschmidt @The-Compiler ! I would like to work on this issue.
Please read the existing comments, we're going in circles here at this point. There is already an open PR for this.