Feature request: $ref from local file

[tomercagan]

First of - very useful library/tool - thanks for making it!

I have a set of OpenAPI files (from these 3GPP specification) which I want to use.

In these files, many $refs point to components from local files.

For example:

'400':
    $ref: 'TS29571_CommonData.yaml#/components/responses/400'

or

dnais:
    type: array
    items:
        $ref: 'TS29571_CommonData.yaml#/components/schemas/Dnai'
    minItems: 1

I see that the generator supports remote references but it does not support local files. It won't be too hard to search-and-replace these references - assuming the files are hosted somewhere, or you want to make the effort to host them yourself, which is not always the case... so maybe it would be good to support such local references.

I've look into the code and it seems a few changes are required.

In this library, in get_ref_body (fastapi_code_generator/parser.py, in line 40) - need to add another case to identify references to a file. I manually tested the following code:

RE_FILE_REF: Pattern[str] = re.compile(r"^.*\.ya?ml#") # at the top

    elif RE_FILE_REF.match(ref):   
        # a new case - get ref body from local file
        filename, path = ref.rsplit('#/', 1)
        ref_body = openapi_model_parser._get_ref_body(filename)
        return get_model_by_path(ref_body, path.split('/'))

I thought this might be enough, and it does work if your cwd is where the files are located. Otherwise, _get_ref_body_from_remote fails because of the way it is currently implemented.

I tried to understand what would be the right approach from here but I am not sure - I couldn't find (yet) a reference to the original file available in parsing another - with the original file name, the $ref-ed file could either be in cwd or relative to the original file.

I sort of got the project to run locally on my machine but I run into issues. I am not sure how quickly I can get everything to run (tests etc).

I'd be happy to help and I will try to setup my computer for development...

Thanks!

Tomer

[tomercagan]

If anyone run into such issue before it was otherwise resolved, here are a few suggestions of how to get this working:

Using remotely hosted copy If using a standardized OpenAPI specifications, you may be able to find it online and the tool knows how to deal with that.

Modify and host your definitions files This will require to search-and-replace all the $refs to start with http/https.

• Find a simple server that can serve a directory, run it and note what address it serves when running (should be something like http://localhost:8080/mydir). Python has such a built-in server if I recall correctly.
• Update your $refs to include that address. I.e. if the reference was 'somefile.yaml#/component/schema/whatever' is should now be 'http://localhost:8080/mydir/somefile.yaml#/component/schema/whatever'
• Move your files to the served directory.
• Run fastapi-codegen

Make the change above and run from specifications directory

This is a little more involved, the steps should be somewhat like:

• Install poetry
• Clone the repository
• cd to the repository location and run poetry install
• Modify the file parers.py as per above, this should now support local files
• cd to the specification file location and run the fastapi-codegen command.

(I believe it should work)

Use OpenAPITools/openapi-generator for generating a single specs file and then run fastapi-codegen tool

OpenAPITools has a generator that can generate code for some other frameworks' (i.e. flask but many more). As part of the generation, it also creates a single yaml file with all the dependencies together. This file is used for serving the documentation). You can run this openapi-generator, locate the "openapi.yaml" file and then use it with fastapi-codegen:

• Install openAPITools generator - see here
• Run the command to generate a flask application. I believe you have to do it from folder there specification files are. I used:

java -jar /root/bin/openapitools/openapi-generator-cli-5.0.0.jar generate -i your_specs_file.yaml -o output_dir -g python-flask

• Find the location of openapi.yaml file, it should be in *output_dir/openapi_server/openapi/
• Run fastapi-codegen on this file. This should create the relevant main.py / model.py

[koxudaxi]

@tomercagan Thank you for creating this issue and giving workarounds 😄

I explain why this project hasn't supported $ref from the local file yet. I'm working on developing datamodel-code-generator aggressively. A lot of user uses datamodel-code-generator that to generate models from JSONSchema/OpenAPI, etc... They request resolving complex $ref and modular models. I clear this problem and request step by step. I will bring the feature into the fastapi-code-generator after I implement these features in the datamodel-code-generator.

For example, I want to support HTTP/HTTPS file directly. https://github.com/koxudaxi/datamodel-code-generator/issues/289#issuecomment-751488925

However, I always welcome any PR. If we can keep unittest then, I accept the PR. Because This project is used in production. We have to keep the same output from the same input file.

[tomercagan]

@koxudaxi - I have created a patch to add local ref support along with relevant tests.

When I run the unit tests (./scripts/test.sh) I am getting a failure in one of the existing tests:

FAILED tests/test_generate.py::test_generate_custom_security_template[oas_file0] - 
AssertionError: assert '# generated ...:\n    pas...

Initially I thought I broke something but this happens (to me) on a clean clone without my changes...

I have have a commit ready to be PRd but not sure if I should proceed because of this (seemingly unrelated) failure

[koxudaxi]

@tomercagan I feel the change is good as the first step 🎉 But, I guess pets.yaml#/components/parameters/MyParam is not parsed in pets.yaml. If you get MyParam then the parser parsed it from body_and_parameters.yaml openapi_model_parser._get_ref_body is only get dict object from openapi. The method parses any object. We have to parse target object with openapi_model_parser.parse_xxxx method. I'm thinking of implementation to resolve the problem 🤔

[tomercagan]

@koxudaxi - I am not sure I understand your comment...

When I remove the code I added and run the new tests, I am getting

E NotImplementedError: ref=pets.yaml#/components/parameters/MyParam is not supported

Which means it never continue past the local reference. The "negative" test I added also shows the same - if pets.yaml is not in proper path relative to pwd, it throws an error.

With that additional code I made, parsing is completed successfully.

I also removed components/parameters/MyParam from body_and_parameters.yaml altogether - I should have done it before... so now the correctness should be more clear.

As for openapi_model_parser._get_ref_body being a generic function (if that what your intension was) that returns a dictionary - I've seen it used in the code so I continue using it... it seems to work just fine :-)

I hope this clears it up a little bit. If I can somehow help - let me know...

[koxudaxi]

@tomercagan I'm sorry for my comment is wrong. I wanted to say you get an error when you remove components/schemas/PetForm in body_and_parameters.yaml.

I cloned your repo and I tested it with the changes.

Changes

diff --git a/tests/data/openapi/local_ref/body_and_parameters.yaml b/tests/data/openapi/local_ref/body_and_parameters.yaml
index 3ce85aa..ee97437 100644
--- a/tests/data/openapi/local_ref/body_and_parameters.yaml
+++ b/tests/data/openapi/local_ref/body_and_parameters.yaml
@@ -219,7 +219,7 @@ components:
       content:
         application/json:
           schema:
-            $ref: '#/components/schemas/PetForm'
+            $ref: 'pets.yaml#/components/schemas/PetForm'
   securitySchemes:
     BearerAuth:
       type: http
@@ -247,11 +247,3 @@ components:
           format: int32
         message:
           type: string
-    PetForm:
-      title: PetForm
-      type: object
-      properties:
-        name:
-          type: string
-        age:
-          type: integer

Test results

...
>               assert output_file.read_text() == expected_file.read_text()
E               AssertionError: assert '# generated ...essage: str\n' == '# generated ...int] = None\n'
E                 Skipping 336 identical leading characters in diff, use -v to show
E                   ssage: str
E                 - 
E                 - 
E                 - class PetForm(BaseModel):
E                 -     name: Optional[str] = None
E                 -     age: Optional[int] = None

/Users/koudai/repo/fastapi-code-generator/tests/test_generate.py:117: AssertionError
...

[tomercagan]

@koxudaxi - you are correct - first, my input files are a little sloppy - I didn't remove duplicate stuff and next, the test indeed fails when I do so.

I tried to look into what is going on and I can't say I have good enough grasp to pinpoint the issue. I will say that the main.py assertion is ok and the problem comes with the model file which from what I gather is completely generated by datamodel_code_generator which I guess entails a deeper investigation into that...

I actually started to look into datamodel_code_generator tracking another bug/error/omission but I am not sure when I will have enough time to "go deep"...

[tomercagan]

I thought about it more and I am not sure why the main file is generated properly while the model does not - in general they rely on the same parser... I am wondering/considering that maybe first we have to do some work on datamodel_code_generator - but from what I understood - it is a more sensitive project...

[koxudaxi]

There are two roles in the codegen.

1. generate main.py by fastapi_code_generator.parser.OpenAPIParser
2. generate models.py by datamodel_code_generator.parser.openapi.OpenAPIParser aka fastapi_code_generator.parser.OpenAPIModelParser

1. generate main.py

You have implemented a parser for main.py. This parser searches simple type(like str, int, float...) and referenced model($ref). The parser doesn't deep dive into referenced model. It gets only a model name.

2. generate models.py

models.py walks into the OpenAPI file to generate models completely.

I thought about our problem. datamodel-code-generator only parses definitions in the input OpenAPI file. We need to run this parser for PetForm in pets.yaml. It mean we should run OpenAPIModelParser.parse_raw_obj().

it is a more sensitive project...

Also, the datamodel-code-generator is used by a lot of users(31k download/month). We can't break interfaces for users. But, I'm working on adding features and fixing bugs. This project is an experimental phase. the version is 0.x.x I will refactor this project by user requests. I hope the fastapi-code-generator walks in the same way.

I will think of the specification when I get time.

Thank you.

[harryjubb]

Hello, is there any traction on this?