godotengine
Exported dictionaries have their keys sorted alphabetically
Tested versions
- Tested with Godot 4.1.2 and Godot 4.3 dev 5
System information
Godot v4.3.dev5.mono - Windows 10.0.22631 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3070 Ti (NVIDIA; 31.0.15.4592) - AMD Ryzen 7 5800X 8-Core Processor (16 Threads)
Issue description
When an exported dictionary is created, they keys which are set in the editor, are sorted at runtime, which is incredibly frustrating for my use case.
Steps to reproduce
- Create an exported dictionary (
@export var dict := {}) - Add some values into the dictionary through the editor. - ({"b": 0, "c": 1, "a": 2})
- Print the dictionary, it will output: "{"a": 2, "b": 0, "c": 1}"
Minimal reproduction project (MRP)
N/A
This is by design, for stability, as far as I know
See also:
- https://github.com/godotengine/godot/issues/74941
- https://github.com/godotengine/godot/issues/54648
should it happen with exported dictionaries though? I get why it needs to happen for dictionaries written to files and such, but i dont see why it should need to be the same with exported dictionaries?
That's what exported dictionaries are though? They are stored and loaded
but why should it be? Other than for stability as you mentioned, which seems only needed for file reading, i dont see any other reason why exported dictionaries should be sorted without the users knowledge. sort_custom should be used instead if the user or the engine REALLY has to have the dictionary sorted
But they are exported, so they're stored, so they're sorted
Are you printing it from the editor? Or from running project? If it's from the running project you are loading it, you have to load the scene?
im running it from the project, but why then can it print the dictionary if its a hard coded variable, even a constant dosent have this issue, and wouldnt that be stored too?
Keeping insertion order is a feature according to the documentation:
Dictionaries will preserve the insertion order when adding new entries.
I don't think it should be sorted during serialization.
Dictionary keeping insertion order is a result of 4.0 changing the implementation to use HashMap instead of OrderedHashMap. But the decision of sorting keys during serialization was made way before that. So I think it's a bug.
The cause of this issue is this line: https://github.com/godotengine/godot/blob/master/core/variant/variant_parser.cpp#L2248. I tried tracking down the original commit, but couldn't quite make it far back enough.
It definitely feels... load-bearing, but I removed it on my build a few weeks ago, and it has even fixed similar issues as well (one of my exported dictionaries kept re-sorting its keys on every resource save, which would have caused endless merge conflicts for my team).
I assume making this change would break existing test cases, but I agree that sorting dictionary keys at this stage of serialization should be considered a bug.
Is there any workaround to this beyond recompiling? JSON-stringify I think loses a lot of information? I guess to use var_to_bytes, which is IIRC stable/in conformance with the documentation, but sacrificing human-readability/diffability? (I was working on an editor for my game, relying on the ordered behaviour of the Dictionaries for displaying properties in the object inspector, but the fields are shuffled about when I load them)
I'm currently touching Dictionary code and want to add some points. This feature is likely to have a significant effect on performance, especially after #104047 is merged. Sorting means we need to copy the keys, sort them, and then perform hashmap lookups one by one. Without sorting, we can iterate directly over the internal HashMap with zero copies. Large Dictionarys (store in JSON or other serialization formats) are quite common in certain types of games (e.g. for storing dialogues or item data). I think it is a trade-off worth considering.
I am reluctant to deem this a "bug" given it's intentionally coded this way, but I've read enough of the above to deem this more annoying than it's worth. It's the only Variant type that does (or can do) this and it breaks user expectations.
I had patched this issue locally for around 3 months and noticed no issues -- of course this is anecdotal and only covered my use cases, but it's enough for me to suggest stripping it for the 4.5 cycle, just to see if anything comes up
no one has a easy way to keep dictionarys from sorting when saving a scene?
I had patched this issue locally for around 3 months and noticed no issues -- of course this is anecdotal and only covered my use cases, but it's enough for me to suggest stripping it for the 4.5 cycle, just to see if anything comes up
As far as I have seen, no one has opened a PR addressing this issue yet. You could be the one.
I had patched this issue locally for around 3 months and noticed no issues -- of course this is anecdotal and only covered my use cases, but it's enough for me to suggest stripping it for the 4.5 cycle, just to see if anything comes up
As far as I have seen, no one has opened a PR addressing this issue yet. You could be the one.
Okie Dokie
As of Godot 4.4 with PR #77213, it is possible to manually sort when needed. In theory this means that it's not needed to do it automatically when serializing anymore, since if the user wants sorting, they can just call the function before serializing.
However, in the past, str_to_var(var_to_str(dict)) was suggested as a hack to sort Dictionaries. Making this change could break projects that expect this to work. Since sort() isn't in Godot 4.3, if PR #104164 was merged for 4.5, this would leave 4.4 as the only "transition period" version where both techniques work. I think ultimately this change is a good one, it's better to keep user data as-is when serializing, but I'm unsure about the timing of merging this.
one of my exported dictionaries kept re-sorting its keys on every resource save, which would have caused endless merge conflicts for my team
I strongly agreed, I haven't made reproducible project or something yet, but it's definitely happend in my two machines that working my same project on editor. (Two machines are shown different order when loaded)
That's why I was keep in eyes on #104503
As of Godot 4.4 with PR #77213, it is possible to manually sort when needed. In theory this means that it's not needed to do it automatically when serializing anymore, since if the user wants sorting, they can just call the function before serializing.
However, in the past,
str_to_var(var_to_str(dict))was suggested as a hack to sort Dictionaries. Making this change could break projects that expect this to work. Sincesort()isn't in Godot 4.3, if PR #104164 was merged for 4.5, this would leave 4.4 as the only "transition period" version where both techniques work. I think ultimately this change is a good one, it's better to keep user data as-is when serializing, but I'm unsure about the timing of merging this.
This is an important caveat to keep in mind indeed!
Perhaps we could have a 'transition period' where we WARN_PRINT_ONCE on var_to_str(dict) if the sorted dict is different than the unsorted dict, to note that relying on auto-sort is deprecated, and will be changed in the future? This should allow everyone whom it concerns to prepare for this eventuality by adding dict.sort() manually before the call.
(and everyone that doesn't care should just ignore the comment)
In my 4.4/4.5-rc1 project, I was relying on Dictionary preserving insert order as stated in the docs, so the fact that serializing dictionaries does autosort was an unexpected behavior that breaks things in my project.
I ended up having to do a hack like this: config_file.set_value(section, "schema", str(schema)), since unlike var_to_str, str does preserve order, and seems to work OK in my current use-case.
I believe behavior should be consistent; when Dictionary states in the docs it preserves insert order, serialization should too. At the very least the caveat needs to be documented, so it does not come as a surprise that insert order is not actually preserved when serializing...
When an exported dictionary is created, they keys which are set in the editor, are sorted at runtime, which is incredibly frustrating for my use case.
Can you please explain your use case?
I think it's wonderful to have dictionaries sorted alphabetically in the inspector. I think things would be much less intuitive for most users if it were random, or just stuck with whatever the actual item addition order happened to be.
Edit:
Oh, so the insert order of the dictionary is being changed, while documentation writes otherwise? I didn't realize this was supported behavior for a dictionary.
Dictionaries will preserve the insertion order when adding new entries.
My comment here is really just concerned about the Inspector's display. If this issue will remove alphabetization, then I would request that the user has some other way to organize an exported dictionary for their Inspector.
