pyyaml
pyyaml copied to clipboard
yaml
Support for the YAML 1.2 Core and JSON schemas [Take 2]
Supersedes #512
This is a draft and subject to discussion. See also https://github.com/yaml/pyyaml/issues/486
(For #512: Thanks to @SUSE for another hackweek! I had four days of work time dedicated to an open source project of my choice. https://hackweek.suse.com/20/projects/yaml-1-dot-2-schema-support-for-pyyaml) Thanks to @SUSE for a volunteer day I used to make this continuation to my previous PR.
This PR depends on #483
Introduction
For a quick overview of the schema changes between YAML 1.1 and 1.2, look here: https://perlpunk.github.io/yaml-test-schema/schemas.html
While also the syntax was changed in YAML 1.2, this pull request is about the schema changes.
As an example, in 1.1, Y, yes, NO, on etc. are resolved as booleans in 1.1.
This sounds convenient, but also means that all these 22 different strings must be quoted if they are not meant as booleans. A very common obstacle is the country code for Norway, NO ("Norway Problem").
In YAML 1.2 this was improved by reducing the list of boolean representations.
Also other types have been improved. The 1.1 regular expression for float allows . and ._ as floats, although there isn't a single digit in these strings.
While the 1.2 Core Schema, the recommended default for 1.2, still allows a few variations (true, True and TRUE, etc.), the 1.2 JSON Schema is there to match JSON behaviour regarding types, so it allows only true and false.
Current State
PyYAML implements the 1.1 types (with a few changes like leaving out the single character booleans y, Y etc.), and it was never updated to support one of the 1.2 Schemas.
Problem
Besides the above mentioned problems with the 1.1 types, more and more libraries are created or updated for YAML 1.2, probably also thanks to the relatively new YAML Test Suite, and PyYAML should be able to read and write YAML files used or produced by other libraries.
This PR
The PyYAML Safeloader, which is currently the most recommended Loader if you don't need special behaviour, implements YAML 1.1 types.
I added tagsets for yaml11, json, core. This way people can try out a YAML 1.2 Loader with little code:
class MyCoreLoader(yaml.BaseLoader): pass
class MyCoreDumper(yaml.CommonDumper): pass
MyCoreLoader.init_tags('core')
MyCoreDumper.init_tags('core')
yaml.load(y, Loader=MyCoreLoader)
Out of Scope
One problem is that PyYAML's callbacks are class based, and while I was able to make the code a bit more compact via a dictionary of types/callbacks, there are still method calls which must be in a certain class.
The !!merge << key for example needs special handling.
That way it's tedious to add custom Loaders. Turning the class based approach into an instance based is on our wishlist.
One example use case we have in mind is, that you want to use the 1.2 CoreLoader, but on top of that you want it to recognize timestamps and mergekeys. Or you want a very basic loader that should treat everything as a string except booleans and null.
Example
import yaml
class MyCoreLoader(yaml.BaseLoader): pass
class MyJSONLoader(yaml.BaseLoader): pass
class MyCoreDumper(yaml.CommonDumper): pass
class MyJSONDumper(yaml.CommonDumper): pass
MyCoreLoader.init_tags('core')
MyJSONLoader.init_tags('json')
MyCoreDumper.init_tags('core')
MyJSONDumper.init_tags('json')
input = """
- TRUE
- yes
- ~
- true
#- .inf
#- 23
#- #empty
#- !!str #empty
#- 010
#- 0o10
#- 0b100
#- 0x20
#- -0x20
#- 1_000
#- 3:14
#- 0011
#- +0
#- 0001.23
#- !!str +0.3e3
#- +0.3e3
#- &x foo
#- *x
#- 1e27
#- 1x+27
"""
print('--------------------------------------------- BaseLoader')
data = yaml.load(input, Loader=yaml.BaseLoader)
print(data)
print('--------------------------------------------- SafeLoader')
data = yaml.load(input, Loader=yaml.SafeLoader)
print(data)
print('--------------------------------------------- CoreLoader')
data = yaml.load(input, Loader=MyCoreLoader)
print(data)
print('--------------------------------------------- JSONLoader')
data = yaml.load(input, Loader=MyJSONLoader)
print(data)
print('--------------------------------------------- SafeDumper')
out = yaml.dump(data, Dumper=yaml.SafeDumper)
print(out)
print('--------------------------------------------- MyCoreDumper')
out = yaml.dump(data, Dumper=MyCoreDumper)
print(out)
print('--------------------------------------------- MyJSONDumper')
out = yaml.dump(data, Dumper=MyJSONDumper)
print(out)
@ssbarnea I'll bring up the task of putting out a new pyyaml release, with the release team.
I suspect this would be merged into the next release.
@ingydotnet 6.0.1 is out, can we have an eta on the next major/minor release that'll merge this pr?
@ingydotnet Is that November target still realistic? Based on the progress so far I becoming quite pessimistic... Now working on replacing most of pyyaml use with ruamel.yaml in ansible-lint.
What worries me the most is that the change was never merged as such code change is also expected to need some time to mature before the release, preferably involving some pre-releases. Still, until is merged to development branch, there is no hope that code would be improved, and instead more likely to be affected by code rot.
@ssbarnea The first week of November is the time that we all agreed to look into putting out the next release. That is when we expect to look at tickets like this.
@ingydotnet May I ask for clarification on the schedule? If my understandings are correct, this looks like to be the current schedule:
- 2023/Nov/03 (end of 1st week of November) Discuss the potential for merging this PR
- 2023/Nov - (some day) Resolve the merge conflict, people can test on an experimental branch
- (some other day) Next release, including this PR
The contents of this PR are included in #700; we've spent much of this past week iterating on that locally in preparation for an upcoming PyYAML 7.0.0a1 in the next couple of weeks (an update/replacement to #700 should happen today with the recent changes).
