feat: Implement json-jsonata manager that can extract dependencies json configuration files

[dariocc]

Changes

Implements a generic json-jsonata manager (https://github.com/renovatebot/renovate/issues/15193) that is able to extract dependencies from json files.

{
  "jsonataManagers": [
    {
      "fileMatch": ["<file match pattern>"],
      "matchQueries": ['<query>'],
      ...
    }
  ]
}

Example configuration file:

{
  "production": [
    {
      "version": "1.2.3",
      "package": "foo"
    }
  ],
  "development": [
    {
      "version": "4.5.6",
      "package": "bar"
    }
  ]
}

Query to extract dependencies:

production.{ "depName": package, "currentValue": version }

Context

Covered in https://github.com/renovatebot/renovate/issues/15193.

Note to reviewers:

• This work is based on the modifications we keep at work from our own fork of Renovate. There it was important to minimize conflicts when syncing with upstream and duplication is a lesser problem. As consequence there is some obvious code duplication because regex managers are special and handling is spread across different places.
• It may be possible to generalize the notion of a custom manager to cover _any sort of "custom" manager. This way renovate would be able to provide a regex manager, a jsonata manager, a jq manager, jsonpath manger ... you name it... and perhaps keep manager validation and specific types within their corresponding manager module. You'll notice that some types such as CustomExtractConfig do not make lot of sense because now there are different custom configs.
• The notion of strategy is not useful for this manager because the JSONata language is complete enough to be able to handle any json document into an array of dependencies. The readability of the expression is however dependent on each particular case and it feels that sometimes it would be useful to apply a final regex transformation to the extracted dependency fields. But I didn't want to over-complicate this work.
• As for support for applying this to yaml it would be possible to parse configuration files using different deserializers (json & yaml), chosen based on config file extension or(optionally) enumeration option under jsonataManagers entries. While this is simple to implement, it can be done in a separate PR if this one is accepted.
• When I did the core of this work, I considered jq and JSONPath. For jq (which would've been my preference due to it's ubiquity) I didn't find any maintained library that I could use. For JSONPath I believe the reason was that it was great selecting but not so much transforming. JSONata was maintained, stable, documented, had no external dependency and looked well thought and implemented. But you'll notice that there are just 2 calls to this library code, so if you find a better fit go for it :).

Documentation (please check one with an [x])

• [x] I have updated the documentation, or
• [ ] No documentation update is required

How I've tested my work (please tick one)

I have verified these changes via:

• [ ] Code inspection only, or
• [x] Newly added/modified unit tests, or
• [ ] No unit tests but ran on a real repository, or
• [x] Both unit tests + ran on a real repository

Tested manager in https://github.com/dariocc/renovate-test.

renovate.json:

  "jsonataManagers": [
    {
      "fileMatch": ["renovate-me.json"],
      "matchQueries": ["production.{ \"depName\": package, \"currentValue\": version }"],
      "datasourceTemplate": "nuget"
    }
  ]

renovate-me.json:

{
  "production": [
    {
      "version": "1.0.0",
      "package": "fenixlib"
    }
  ]
}

PR opened by renovate:

[dariocc]

The PR is in draft until it is tested against a real repo.

[rarkins]

I would like to use a name like "custom" manager for this and make sure it's designed to anticipate future expansions such as YAML and TOML parsing. Ideally it could even aborb the regex manager too if that makes sense.

[dariocc]

The rationale between the chosen name for the manager, or better said the configuration option is that jsonata is to this manager what regex is to the Regex manager.

and make sure it's designed to anticipate future expansions such as YAML and TOML parsing. Ideally it could even aborb the regex manager too if that makes sense.

This is mentioned in the PR description. A problem is that I'm not sure I have enough context to make good decisions. For example how comfortable you'd feel with changing configuration format (for example, if regex manager is removed)?

So I'd appreciate input before doing any further modifications.

What would you think about the following alternative:

{
  "customManagers": [
    {
      "type": jsonata // <regex> | jsonata ....
      ...
    }
  ]
}

Different types of custom manager will have different set of options, that make sense for that specific manager.

For the jsonata type, to handle multiple config file formats we could have something like:

{
  "customManagers": [
    {
      "type": "jsonata",
      "fileFormat": "json" // <auto> | json | yaml | toml
      ...
    }
  ]
}

The regexManager configuration option should then be "deprecated" and a warning should be emitted to migrate to the customManager form. There would also be a bunch of work to do in term on documentation, I assume, and I wonder if someone would be able to contribute on that front.

Would this make sense to you?

[rarkins]

I think this sounds good.

I don't think we need to completely the unification with regex manager in this PR, or handle any migrating. It's just important to foresee how that would go so that we could do it in a follow-up PR or two. Changing configuration format would be fine as long as there's a deterministic mapping.

Small nit: in general I prefer to avoid option names like type within nested options if we can. We can probably think of a better term as this PR matures.

[dariocc]

Yes, I agree to your observation of the the type naming. I just couldn't come up with a better name to indicate that we'll need to discriminate the type of the custom managers.

[dariocc]

I've now tested this PR against a real repository: https://github.com/dariocc/renovate-test. You can see the generated PR.

[dariocc]

I don't think we need to completely the unification with regex manager in this PR, or handle any migrating.

So is it fine to merge this PR as it is and work on the "unification" in a separate PR?

[rarkins]

So is it fine to merge this PR as it is and work on the "unification" in a separate PR?

The names would need changing, so it's better that we get the name right first time, even if it's not unified initially. i.e. call it "custom" manager from the start

[dariocc]

Hi! It has been a while... but I'm finding time again to work on this PR. I was wondering if we can make some decisions so as this reaches a minimum mergeable state...

My understanding is that the following is needed (to avoid future breaking changes):

1. Prepare the options to suport customManagers instead of jsonManager.

Even if we use customManagers, to me it makes sense to keep json-jsonata module because that will still be only able to handle the jsonata custom manager.

2. Decide upon a field name that act as discriminator for the customManagers subtypes, and initially allow only a value of jsonata.

What would you think about the name engine?

{
  "customManagers": [
    {
      "engine": "jsonata",
      ...
    }
  ]
}

In my opinion, support for processing non json document can be implemented later on (the fileFormat field suggested in https://github.com/renovatebot/renovate/pull/17077#issuecomment-1213642788).

[rarkins]

We'll make this migration first: #19066

After that we can add jsonata as a new customType

[rarkins]

@RahulGautamSingh is the customManagers renaming/refactoring now complete and unblocking this one?

[RahulGautamSingh]

@RahulGautamSingh is the customManagers renaming/refactoring now complete and unblocking this one?

We can proceed with this.

[dariocc]

@RahulGautamSingh is the customManagers renaming/refactoring now complete and unblocking this one?

We can proceed with this.

That's great news.

What is the expectation here? Will someone pick up on my work or are you hoping I'll be able to continue it?

[RahulGautamSingh]

@RahulGautamSingh is the customManagers renaming/refactoring now complete and unblocking this one?

We can proceed with this.

That's great news.

What is the expectation here? Will someone pick up on my work or are you hoping I'll be able to continue it?

If you can continue it that will be great since you have done so much work here. Else, I can pick it up.

[dariocc]

If you can continue it that will be great since you have done so much work here. Else, I can pick it up.

I'll see what I can do, but I might need some time.

[HonkingGoose]

Hi @dariocc 👋

Do you have time to finish this PR, or do you want @RahulGautamSingh to take over? I'm asking because it's been a few months now, and this PR is still the same.