Validating yang structure data

[ahassany]

I've tried to validate the example in RFC 8791 using yanglint but without a success, is yang structures fully supported for data and schema or schema only?

This the examples I tried:

module example-module {
  yang-version 1.1;
  namespace "urn:example:example-module";
  prefix exm;

  import ietf-yang-structure-ext {
    prefix sx;
  }

  sx:structure address-book {
    list address {
      key "last first";
      leaf last {
        type string;
      }
      leaf first {
        type string;
      }
      leaf street {
        type string;
      }
      leaf city {
        type string;
      }
      leaf state {
        type string;
      }
    }
  }
}

and input data:

{
  "example-module:address-book": {
    "example-module:address": [
      {
        "city": "Bedrock",
        "example-module-aug:zipcode": "70777",
        "first": "Fred",
        "last": "Flintstone",
        "street": "301 Cobblestone Way"
      },
      {
        "city": "Bedrock",
        "example-module-aug:zipcode": "70777",
        "first": "Charlie",
        "last": "Root",
        "street": "4711 Cobblestone Way"
      }
    ]
  }
}

witht the command:

yanglint  example-module.yang  example.json -f json

libyang err : Node "address-book" not found in the "example-module" module. (line 2)
YANGLINT[E]: Failed to parse input data file "example.json".

[michalvasko]

It seems yanglint(1) does not support this but libyang does. You would have to call lyd_parse_ext_data() yourself in a small app. We may add the support for this in yanglint later.

[ahassany]

Thank you @michalvasko, I will check with my team if we can allocate capacity to contribute back to yanglint for this feature.

[mxyns]

Hi @michalvasko, I'm not a Yang expert, first time having a look at libyang and yanglint. I've had a look at the state of Yang structure extension in both.

The RFC8791 for the Yang data structure extension allows for structures to be at any level of the tree

There is no assumption that a YANG
   data structure can only be used as a top-level abstraction, and it
   may also be nested within some other data structure.

It seems like structures are supported in libyang as a top-level node only structure.c@63: /* structure can appear only at the top level of a YANG module or submodule */ am I understanding that correctly?

It looks like if you want to validate using an extension (here a structure), you need to tell the json/xml/... parser which extension instance (singular) to use to parse & validate before the file is even read. This is because the api for "normal" parsing is

LIBYANG_API_DECL LY_ERR lyd_parse_ext_data(const struct lysc_ext_instance *ext, struct lyd_node *parent, struct ly_in *in,
        LYD_FORMAT format, uint32_t parse_options, uint32_t validate_options, struct lyd_node **tree);

while the api for "normal" w/o extension parsing is

static LY_ERR
lyd_parse(const struct ly_ctx *ctx, const struct lysc_ext_instance *ext, struct lyd_node *parent, struct lyd_node **first_p,
        struct ly_in *in, LYD_FORMAT format, uint32_t parse_opts, uint32_t val_opts, struct lyd_node **op);

It seems like "normal" parsing you just pass the context, and the json/xml parser infers the module to parse with from there using the context. It also looks like the parsing w/ extension requires you to pass the extension instance directly (no inference here).

So, from what I understood of the yanglint code, there is currently no way of doing the validation of a structure with yanglint at the moment.

Assuming, the structure is always the top-level, and is the only structure of the tree, one way to fix that would be to make the user pass as CLI parameter the exact module name to validate the data against, on top of the list of modules to load. Something like yanglint module-1.yang module-2.yang message.json --top-level module-1.yang. From there we can programmatically find the exact extension instance used for the top-level and pass it to the parser.

If this is not acceptable, we would need to change the parsers so that they first determine if the top-level node uses an extension or not, and parse based on that.

Am I on the right track here? If so, is the first idea of adding a parameter to yanglint acceptable?

[michalvasko]

We actually have a student working on this but it may take some time.

Regarding the nested structure support, I am confused because in the YANG module there is

          This extension is only valid as a top-level statement, i.e.,
          given as a substatement to 'module' or 'submodule'.

Also, you are correct that yanglint will need some parameters for top-level structures, something like --yang-structure <name>, you need to know both that it is a structure and which exactly.

[mxyns]

Thank you @michalvasko

Regarding the nested structure support, I am confused because in the YANG module there is

    This extension is only valid as a top-level statement, i.e.,
    given as a substatement to 'module' or 'submodule'.

That is indeed confusing 😅 I've asked a colleague of mine. To them, this means that the structure has to be top-level in a (sub-)module definition. However, that structure might not be top-level in whatever data you try to validate. That can be the case if the data you have has a structure contained in an anydata node of another module. This makes sense to me.

This would mean that we need to change the parsers to detect if any node of the tree is a structure and not only at top-level. Maybe we can begin with just the --yang-structure <module-name.yang> and enhance that later?

[mxyns]

Also, I'm not sure just the structure name or just the module name is enough. We also need the name of the module in which the structure is defined since multiple modules can define a structure with identical names, right?

[michalvasko]

I have asked about the possible placement of structure data on the netmod mailing list. But regardless of that, the feature will not be changed for now and the yanglint support is actually almost implemented and should be merged into devel today. I will reply to this issue once it is.

[mxyns]

okay that's great thanks

[mxyns]

linking the PR to the issue for future reference #2410

[michalvasko]

PR has been merged, you can test the feature.

[mxyns]

hi @michalvasko @Ardival i'm trying to validate the example json given by @ahassany with the example-module as described in the updated README.md.

I've removed the structure augmentation from example-module-aug for now so which means I'm validating example.json:

{
  "example-module:address-book": {
    "example-module:address": [
      {
        "city": "Bedrock",
        "first": "Fred",
        "last": "Flintstone",
        "street": "301 Cobblestone Way"
      },
      {
        "city": "Bedrock",
        "first": "Charlie",
        "last": "Root",
        "street": "4711 Cobblestone Way"
      }
    ]
  }
}

with example-module.yang:

module example-module {
  yang-version 1.1;
  namespace "urn:example:example-module";
  prefix exm;

  import ietf-yang-structure-ext {
    prefix sx;
  }

  sx:structure address-book {
    list address {
      key "last first";
      leaf last {
        type string;
      }
      leaf first {
        type string;
      }
      leaf street {
        type string;
      }
      leaf city {
        type string;
      }
      leaf state {
        type string;
      }
    }
  }
}

here is what i get

max@max-tk:~/CLionProjects/libyang$ yanglint 
> clear
> add /home/max/yang/example-module.yang
> list
List of the loaded models:
    i ietf-yang-metadata@2016-08-05
    I yang@2025-01-29
    i ietf-inet-types@2013-07-15
    i ietf-yang-types@2013-07-15
    I ietf-yang-schema-mount@2019-01-14
    i ietf-yang-structure-ext@2020-06-17
    I example-module
> data -t ext -k example-module:structure:address-book /home/max/yang/example.json
YANGLINT[E]: Nothing to validate in the extension input data.
YANGLINT[E]: Failed to parse input data file "/home/max/yang/example.json".
> 

do you see what I'm doing wrong? seems like the json parser returns a NULL tree after the lyd_parse_ext_data but I'm not sure why

[mxyns]

I've also tried with

{
  "example-module:address": [
    {
      "city": "Bedrock",
      "first": "Fred",
      "last": "Flintstone",
      "street": "301 Cobblestone Way"
    },
    {
      "city": "Bedrock",
      "first": "Charlie",
      "last": "Root",
      "street": "4711 Cobblestone Way"
    }
  ]
}

and I get the following error

YANGLINT[E]: Yanglint does not support more than one top-level node in extension data.
YANGLINT[E]: Failed to parse input data file "/home/max/yangshi/example.json".

[michalvasko]

Right, the second one is correct and should now validate.

[ahassany]

Hmm, just asking why the second is valid not the first one?

[michalvasko]

It seems you are right, the first JSON example is correct according to the specification, with the extension itself represented similarly to a container. We have missed that and the implementation of this extension is wrong. However, the definition really made an effort to make things difficult and there is no reasonable way to fix this currently. I will try to come up with something in the following days.

[ahassany]

Thank you @michalvasko for the quick fixes, looking forward for final fix.

Just FYI we have demoed this in IETF hackathon, NMOP, and will do more detailed demo in an IETF side meeting on Friday morning at 9:30 see https://trello.com/c/SEzdB5jS .

[michalvasko]

Maybe too late but I have pushed the fix in a separate branch, referenced. It required major changes and additional functionality of our extension plugin API, for such a minor feature. The authors probably did not realize how big a change compared to the standard YANG behavior it is.

[ahassany]

Thank you, I will refer to it while talking.

[michalvasko]

Just note that some NBC changes were needed, which is why it is in a separate branch. It will be merged into upstream but iteratively and it will take some time.

[ahassany]

just checking any on the refactoring for the NBC change?

[michalvasko]

The change is currently in the so4 branch, which includes several NBC changes. After some final testing it will get merged into devel sometime at the beginning of the next week. Then, in at least 3 months it will be released and merged into master.