Failing to correctly parse AWS CloudFormation Yaml

[godeater]

Hi there,

I'm struggling to use this module to parse an AWS CloudFormation template. The template is mostly strict YAML, but AWS define some "intrinsic functions" which can appear as tag values with a '!'prefix on the function name, followed by its arguments, e.g.

Content:
  S3Bucket: !Ref SomeS3Bucket

or

OtherContent:
  BucketArn: !Sub '{"NodeJsCfnResponseArn":"${LambdaLayerNodeJsCfnResponse}", "NodeJsSendEmailArn":"${LambdaLayerNodeJsSendEmail}"}'

When I try to convert these templates to a PowerShell object using ConvertFrom-Yaml, the function arguments seem to be present in the resulting object, but the function itself has gone completely. Contrived example below:

AWSTemplateFormatVersion: 2010-09-09
Description: A contrived CloudFormation Template
Resources:
  KmsKeyAlias:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: alias/AKMSKey
      TargetKeyId: !Ref AKMSKey

Now run this through the module:

# $Yaml is content as above
$Template = $Yaml | ConvertFrom-Yaml
$Template.Resources.KmsKeyAlias.Properties

Name                           Value
----                           -----
AliasName                      alias/AKMSKey
TargetKeyId                    AKMSKey

Where really I'd prefer to see:

Name                           Value
----                           -----
AliasName                      alias/AKMSKey
TargetKeyId                    !Ref AKMSKey

I tried looking at your docs on adding custom schemas, and apparently that's over my YAML event horizon, as I couldn't make head or tail of it.

Any pointers?

[godeater]

Okay, I think I've got something which works (for me). I did this:

$schema = New-YamlSchema -ParseScalar {
  param($Value, $Schema)
  switch ($Value.Tag) {
    {$_ -eq "!Ref"} {
                      $Out = "{0} {1}" -f $Value.Tag, $Value.Value
                      $Out
                    }
    {$_ -eq "!Sub"} {
                      $Out = "{0} {1}" -f $Value.Tag, $Value.Value
                      $Out
                    }
    default         {
                      $Schema.ParseScalar($Value)
    }
  }
}

# Then using the same yaml as in the original post:
$Template = $Yaml | ConvertFrom-Yaml -Schema $schema
$Template.Resources.KmsKeyAlias.Properties

Name                           Value
----                           -----
AliasName                      alias/AKMSKey
TargetKeyId                    !Ref AKMSKey

And that appears to have worked for my narrow use case. Is it the right approach, or should I have done something different?

[jborean93]

This is certainly the way to get the data the way you would like it right now. Keep in mind the !... is a special YAML syntax to specify the node's tag so trying to then re-seralize the value back to YAML will require a custom emitter to add the tag back in. If you don't do that you'll get a quoted value

$h = @{foo = '!Ref AKMSKey'}
$h | ConvertTo-Yaml

# foo: '!Ref AKMSKey'

The only way to emit a value with an explicit tag is to provider it as a Yayaml.ScalarValue

$h = @{
    foo = [Yayaml.ScalarValue]@{
        Value = 'AKMSKey'
        Tag = '!Ref'
    }
}
$h | ConvertTo-Yaml

# foo: !Ref AKMSKey

It would be nice to preserve the node metadata on the incoming object so you could access it normally without any extra work and also have it emit in the same format. If I find some time I'll try and come up with a good solution there.

[godeater]

Ah, you're right - I did indeed run into that problem when trying to re-serialize back to Yaml!

I tried a bunch of things with EmitScalar on New-YamlSchema and can't believe I didn't try just setting a Tag. I was doing various incantations of .Style = 'Plain' and couldn't make them work - I kept getting the single quoted value as you said above. In the end I commited the gross sin of using a regex replace on the entire output document to remove the quotes from around !Ref and !Sub values.

I've just tried this based on your kind tip above, which seems to do what I want:

$emitSchema = New-YamlSchema -EmitScalar {
  param($Value, $Schema)
  if($Value -match '(?<Tag>!Ref|!Sub) (?<Value>.*)'){
    [Yayaml.ScalarValue]@{
      Value = $Matches.Value
      Tag   = $Matches.Tag
      Style = 'Plain'
    }
  } else {
    $Schema.EmitScalar($Value)
  }
}
$h = { foo = '!Ref AKMSKey' }
$h | ConvertTo-Yaml -Schema $emitSchema
# foo: !Ref AKMSKey

Uses regex again, but at least it's not on the entire document now 😁

At the moment this is sufficient for my current use case, as I'm not dealing with a template which includes any of the more complex intrinsic functions which CloudFormation supports.

[godeater]

A follow up to this, which I can't make work - my template has become a bit more complicated now, and requires use of another AWS intrinsic function '!Join'.

S3Bucket: !Join
  - '-'
  - - !Ref S3BucketName
    - !Ref AWS::Region
    - !Ref AWS::AccountId

I've tried to update the Schema that I was using before when importing the template, which originally looked like this:

$AWSSchema = New-YamlSchema -ParseScalar {
      param($Value, $Schema)
      if($Value.Tag -in @('!GetAtt', '!Ref', '!Sub')){
        "{0} {1}" -f $Value.Tag, $Value.Value
      } else {
        $Schema.ParseScalar($Value)
      }
    }

to this:

$AWSchema = New-YamlSchema -ParseScalar {
  param($Value, $Schema)
      if($Value.Tag -in @('!GetAtt', '!Join', '!Ref', '!Sub')){
        "{0} {1}" -f $Value.Tag, $Value.Value
      } else {
        $Schema.ParseScalar($Value)
      }
    } -ParseSequence {
      param($Values, $Schema)
      $Schema.ParseSequence($Values)
    }

And then doing single-step debugging to try to see what I need to do to make the import work - but at the point where the parser is trying to handle the first line, I'm seeing this in the debugger:

So the parser doesn't appear to be seeing the '!Join' tag at all?

[jborean93]

Sorry for the delay I've been on a break and just got back. The tag should be on the -ParseSequence entry for ['-', {...}] but when debugging as the breakpoint screenshot you have is the key entry rather than the value. It does not appear to be present so it needs some changes to make it available.

Luckily it is present on the underlying YamlDotNet library, I just need to add it to the SequenceValue that is being passed along to the ParseSequence call.

[jborean93]

I've opened https://github.com/jborean93/PowerShell-Yayaml/pull/27 which adds the Tag property to the Parse* map and sequence values. I've got an issue with CI that needs to be fixed but hoping that'll be done by tomorrow so I can push out a new release then.