SafeConstructor.yaml_flatteners, to enable pluggable flattening akin to add_constructors

[ross]

As part of working through https://github.com/octodns/octodns/pull/1315 I wanted to add support for <<: !include some-file.yaml and ran into the same problems as others: https://github.com/yaml/pyyaml/issues/632 and https://github.com/yaml/pyyaml/issues/814.

After a few false starts and unworkable ideas I landed on adding the ability to have pluggable flattening the same way that constructors are now. It's a fairly trivial change in the repo and I'm able to port the existing map and seq blocks over and have all the tests passing.

More interestingly it allows 3rd parties to add new flatteners, so in the case of !include I can do something like:

class IncludingLoader(SafeLoader):

    def construct_include(self, node):
        mark = self.get_mark()
        directory = dirname(mark.name)

        filename = join(directory, self.construct_scalar(node))

        with open(filename, 'r') as fh:
            return load(fh, self.__class__)

    def flatten_include(self, node):
        mark = self.get_mark()
        directory = dirname(mark.name)

        filename = join(directory, self.construct_scalar(node))

        with open(filename, 'r') as fh:
            yield compose(fh, self.__class__).value

IncludingLoader.add_constructor('!include', IncludingLoader.construct_include)
IncludingLoader.add_flattener('!include', IncludingLoader.flatten_include)

In the context of octoDNS that allows config like:

# octodns.yaml
...

_zones: &a
  first.com.:
    sources:
      - config
    targets:
      - out

zones:
  <<: *a

  <<: !include zones.yaml

  primary.com.:
    sources:
      - config
    processors:
      - ownership
    targets:
      - out

# zones.yaml
---
other.com.:
  sources:
    - config
  targets:
    - out

To work exactly as expected.

I'm out of time for now, but I plan to come back to this PR and look at adding specific tests around the ability to add custom flatteners and verify that the more complex versions of <<: work, i.e. arrays and custom flatterers in anchors.

Thoughts and suggestions welcomed.

[ingydotnet]

Not specifically related to this PR, but I'd lean towards !load over !include unless it's already in heavy use. That would be my current direction if we try to standardize around some kind of yaml stdlib.

[ingydotnet]

Well, then again, if pyyaml offered a !load in the future, it wouldn't conflict with a custom !include. :shrug:

[ross]

I have no preference and haven't implemented anything "real" on the octoDNS side yet. Only testing out what I've been playing around with here. I've pushed up a test case for the simplest path. Looking at the details of supporting things like:

foo:
  <<: [!custom ..., !custom, ...]

Now as the current implementation doesn't support them. flatten_yaml_seq assumes each value item in the sequence is a MappingNode and doesn't "flatten" them.

[ross]

Ok. Fixed the sequence case so that it will also handle custom flatteners. Was a trivial change and it actually works much more like map now so that's probably an added benefit. Test cases for the original handling of maps in sequences along with custom flatteners added as well.

Going to move this out of draft as I believe it's now at least to the point where it's ready for :eyes: and :thought_balloon:s

[ingydotnet]

Out of time for today, but I'll try to follow up more tomorrow.