Convert existing rules using jsonpath for reusability across tooling

[jeremyfiel]

Is your feature request related to a problem? Please describe. We have many rules outlined in our own prior work and also from the community at large who are creating some really great ones. The difficulty lies in the reusability from other tooling.

A few examples of some rulesets out in the wild. https://github.com/postman-open-technologies/linting-rules/tree/main/_rules https://github.com/italia/api-oas-checker/tree/97b708582940dececbf983502094b7fca92f2365/rules https://github.com/Azure/azure-api-style-guide/blob/main/spectral.yaml

Describe the solution you'd like

I would love to reuse existing rules and convert them into "custom rules" for openapi-cli. I believe the reason this is not possible is due to the lack of jsonpath support in your implementation, rather it uses the nifty tree traversal.

I would like some way for us to convert those rules into the format required for your tool. The current way of writing custom rules requires knowledge of JS/TS and we have some users who lack fluency in writing code, rather they are familiar with the jsonpath work we've compiled previously.

Describe alternatives you've considered

Short of rewriting them all in the custom rules format, no other alternative has been explored

[RomanHotsiy]

Hey @jeremyfiel, we recently released assertions. Check it out. Maybe it will be a good alternative: https://redocly.com/docs/cli/resources/rules/assertions/

[jeremyfiel]

hey Roman.. the new feature looks great and I'll try it out soon.

The one thing still missing related to my original comment, interoperability with the ecosystem. That's what I'm really looking for.

[RomanHotsiy]

We're not going to support json pointer. I have some thoughts on it. I'll try to summarize it here later this week.

[adamaltman]

@jeremyfiel heads up: I removed the assertions docs yesterday and made this issue #647. We'll release the docs once this change is made.

You raise some good issues, and I'll leave @RomanHotsiy to expand on JSON Path, but I'll give you my take:

1. We need support for reusable configs.
2. We need a place to publish and share configs.
3. We need a mechanism for a wide variety of rules without knowing JS. Unusual rules may require JS. This is where the JSON Path comes into play.

Reusable configs

This is in development now. If all goes as planned (does it ever?), we should have them within a month (the ability to extend from other configs on the file system or possibly remote URLs).

A community

We need a place to publish and share configs. As best I can think of at the moment is a Markdown page with a link to various configs.

A mechanism for wide variety of rules

We have a lot of built-in rules. But we have one super-rule (albeit under construction here #647 ) which is about assertions. It's very flexible and allows you to apply various types of logic to different parts of an OpenAPI definition.

We've gone with the type approach, so you put a subject like Responses, and apply the logic to that subject. This is similar but different from using JSON Path. The type tree is more error resistant. JSON Path is hard to use and understand.

My introduction to JSON Path...

Looking at one of the MAANG rulesets using JSON PATH two years ago

Adam: What does that mean? Roman: It is searching the operation parameters. No, the path item parameters. Hold on... No, I don't think this does what they think it does. Adam: Can we test it? Roman: Yes. ... 5 minutes later ... Roman: Yes, it doesn't do what they think it does.

While the above summarizes my introduction to JSON Path... we have the same type of conversation with every ruleset we look at (including some you linked above).

This is a conversation from three days ago

Roman: Why did they choose to use both products? Adam: They said they chose it for lint because they investigated it a year ago and thought it was more robust. Then, they started to use our bundler because they liked the output better. But I'm looking at their ruleset and I don't understand some of it. What does that actually do? Roman: ...looking... ... 5 minutes later ... Roman: Yes, it doesn't do what they think it does.

What's changed? I no longer need to ask to test it. 🤣 But it doesn't do what people think... Literally. Almost every single time.

Victims of incorrect JSON Path include prominent figures who will remain unnamed besides myself.

[jeremyfiel]

@adamaltman i do agree JSON path+ is an animal on it's own. I've definitely been a victim spending hours on trying to make it do what I want. I really do like the type approach and I think it favors itself to performance much more than other tooling. Looking forward to see what comes of this.

[tatomyr]

I'm closing this issue for now. Feel free to reopen if you think we're missing something.