CLI reports different issues from studio

[petkostas]

Describe the bug I have a set of OAS split specifications while running studio and checking any of the OAS files I get different kinds of linting errors between the studio (local) and spectral cli Even if I fix the errors in the studio, I keep getting some inconsistencies between the CLI and the studio. Additionally, it seems that spectral encounters problems when using properties that are named similar to OAS keywords (in the above example enum which is used as a field name for the API payload). The CLI does not complain about the enum.

To Reproduce

Unfortunately, I cannot share the OAS, but I might try to reproduce it with a generic one.

Expected behavior Spectral CLI and Stoplight studio align in the reported errors.

Screenshots Studio:

CLI:

Environment (remove any that are not applicable):

• Library version: 6.3.0
• OS: Ubuntu Linux
• Browser: not relevant.

Additional context Add any other context about the problem here.

[P0lip]

Hey!

Unfortunately, I cannot share the OAS, but I might try to reproduce it with a generic one.

That'd be great.

The only meaningful difference between Studio and CLI is the way $refs are resolved. Does your document have plenty of $ref?

[petkostas]

Hello, @P0lip to your question yes we use quite a few references within our APIs. I tried to create an example based on the existing specifications and replicate the errors, so this comes a bit close to what is happening. As you can see with the attached files, in stoplight studio I am getting some lint errors that are not present in the cli, most importantly the complaint about enum which in this case is used as an actual payload field. While on the CLI there is no complaint about the enum, but there is a complaint for the provided example of the model: I have attached a sample directory with references that produces the errors: stoplight-test.zip

[P0lip]

Thank you! I'll take a look at it when I have a minute and will get back to you.

[P0lip]

Hey, apologies for the delay. I took a look and everything seems to be working fine.

How did you install Spectral?

[petkostas]

Hello @P0lip sorry for the late reply, I tried both the npx approach, and normal installation and also npm installation and I get similar issues. Another example that came to our attention yday, is the shared models, for example, if you create a shared model/object and add an example to it, Stoplight studio does not report any problems. When you run spectral on your command line, spectral reports problems, which are related to how Stoplight Studio creates the shared model examples. I will come back again with another example, as I can't provide here the actual data (we have also opened a case with the company account towards stoplight, if you would like to know the case number please reach out to me).

[jerryjili]

Has there been any updates to this issue? I ran into the same issue and it was happening on Windows 11. Basically, the EXE version gave different errors from the cli version, although the spectral version number was both 6.6.0. It looks like in the "given" field in the rulesets, they are hitting different depth levels of the yaml file, exactly for the "example"s resolved with $ref.

For example, on the same contract and same ruleset, I have specified the "then" to execute a console log and will print the path the validation is running on. Here are different outputs: Windows EXE version

test hit path paths,/,post,requestBody,content,application/json
test hit path paths,/,get,responses,200,content,application/json,example
test hit path paths,/,post,responses,202,content,application/json,example
test hit path paths,/,post,responses,400,content,application/json,example

c:/****/****/****/some-service.yaml
  11:9   warning  operation-tags              Operation must have non-empty "tags" array.      paths./.get
  61:10  warning  operation-tags              Operation must have non-empty "tags" array.      paths./.post
  68:24  warning  oas3-parameter-description  Parameter objects must have "description".       paths./.post.parameters[0].description
 150:23  warning  oas3-unused-component       Potentially unused component has been detected.  components.schemas.TimestampResponse

CLI version running on node 16.13.2

Linting c:/Users/jili/Downloads/acknowledgement-service-openApiYaml.yaml
test hit path paths,/,get,parameters,0,schema
test hit path paths,/,get,parameters,1,schema
test hit path paths,/,post,parameters,0,schema
test hit path paths,/,post,parameters,1,schema
test hit path paths,/,post,requestBody,content,application/json,schema

c:/****/****/****/some-service.yaml
  11:9   warning  operation-tags                             Operation must have non-empty "tags" array.      paths./.get
  37:23  warning  custom-request-and-response-example-value  Missing example.example value.                   paths./.get.responses[200].content.application/json.example
  61:10  warning  operation-tags                             Operation must have non-empty "tags" array.      paths./.post
  68:24  warning  oas3-parameter-description                 Parameter objects must have "description".       paths./.post.parameters[0].description
 118:23  warning  custom-request-and-response-example-value  Missing example.example value.                   paths./.post.responses[202].content.application/json.example
 132:23  warning  custom-request-and-response-example-value  Missing example.example value.                   paths./.post.responses[400].content.application/json.example
 150:23  warning  oas3-unused-component                      Potentially unused component has been detected.  components.schemas.TimestampResponse
 370:16  warning  oas3-valid-schema-example                  "example" property type must be object           components.schemas.AcknowledgementContract.example

[philsturgeon]

Studio Desktop is gone so this problem has kinda solved itself.