Importing generated code results in a circular dependency

[gal-gocredd]

Describe the bug I'm trying to use datamodel-codegen to generate Pydantic models from Stripe's openapi. The generation itself succeeds but trying to import the generated package results in an import error due to circular dependency:

In [1]: import stripe_api
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 import stripe_api

File .../stripe_api/__init__.py:13, in <module>
      9 from typing import Any, Dict, List, Optional, Union
     11 from pydantic import BaseModel, Field, constr
---> 13 from . import issuing, test_helpers, treasury
     16 class BusinessType(Enum):
     17     company = 'company'

File .../stripe_api/issuing.py:12, in <module>
      8 from typing import Dict, List, Optional, Union
     10 from pydantic import BaseModel, Field, constr
---> 12 from . import (
     13     BalanceTransaction,
     14     IssuingAuthorizationAmountDetails,
     15     IssuingAuthorizationMerchantData,
     16     IssuingAuthorizationPendingRequest,
     17     IssuingAuthorizationRequest,
     18     IssuingAuthorizationTreasury,
     19     IssuingAuthorizationVerificationData,
     20     IssuingCardAuthorizationControls,
     21     IssuingCardholderAddress,
     22     IssuingCardholderAuthorizationControls,
     23     IssuingCardholderCompany,
     24     IssuingCardholderIndividual,
     25     IssuingCardholderRequirements,
     26     IssuingCardShipping,
     27     IssuingCardWallets,
     28     IssuingDisputeEvidence,
     29     IssuingDisputeTreasury,
     30     IssuingTransactionAmountDetails,
     31     IssuingTransactionPurchaseDetails,
     32     IssuingTransactionTreasury,
     33 )
     36 class AuthorizationMethod(Enum):
     37     chip = 'chip'

ImportError: cannot import name 'BalanceTransaction' from partially initialized module 'stripe_api' (most likely due to a circular import) (.../stripe_api/__init__.py)

It looks like the generation puts every model that includes a . for example issuing.authorization in its own file, in this example issuing.py, and every model that doesn't - in the __init__.py file, resulting in some models from the init trying to import files from other files, and those other files trying to import from the init.

Trying to generate everything into one file fails with Modular references require an output directory, not a file.

To Reproduce

Example schema: I tried using both the .yaml and .json just in case, both result in the same error.

Used commandline:

$ datamodel-codegen --input stripe-openapi/openapi/spec3.yaml --output stripe_api

Expected behavior I'd expect to be able to import the generated code

Version:

• OS: macOS Monterey 12.5
• Python version: 3.9.13
• datamodel-code-generator version: 0.13.1

[sarinniural]

@gal-gocredd did you find a workaround for this?

[koxudaxi]

If someone has an idea to resolve the circular dependency structure then could please post it? I don't want to change the structure of generated code. But, I should kick out of the class to another module.

[noddycode]

I just wrote a post-processing script to get around this by moving the imports of other models to the bottom of the script before any calls to update_forward_refs based on an SO answer (I believe relative imports only work this way in 3.5+). Here's a quick mockup I made earlier today to show the difference in behavior.

I've still got some testing to do, but it at least allows me to import modules with circular imports without errors. It did require me to "trick" dataclass-codegen into putting each model in its own file to reliably move all of the imports before that method call.

I would propose:

1. Splitting the imports into 3 sections rather than 2, putting all of the relative model imports in their own instance of Imports
2. Adding an option for whether the imports should go at the top of the file or the bottom (could be some people want to be warned of circular imports)
3. Adding all three import groups to the jinja templates rather than adding them on after the template has been generated to allow greater customization if necessary

Things I still need to look into before I confidently declare this a viable solution:

1. Making sure this still works with multiple models defined in the same file
2. Making sure this still works when models are kept in definition order
3. Trying this out with dataclasses (I used pydantic for my project)

[Yoshify]

@noddycode would you care to share your post-processing script? Your quick mockup link has expired. Unfortunately running into the same issue here after generating models based off the ConnectWise OpenAPI spec.

[noddycode]

@noddycode would you care to share your post-processing script? Your quick mockup link has expired. Unfortunately running into the same issue here after generating models based off the ConnectWise OpenAPI spec.

Unfortunately the code is proprietary, but I can write up a summary of what it does later today.

In the meantime, one thing I would/will eventually change about my script is using the TYPE_CHECKING boolean along with annotations from future instead of moving imports to the bottom. See here:

https://stackoverflow.com/a/72667915

Given some time in the next few weeks, I'd like to try and contribute something that does this during generation.

[aaravind100]

Hey, I'm trying to the create models for ms adaptive cards (schema here) since the python sdk is not available yet. It causes circular imports for this schema too. Any solutions for this?

I ended up creating a thin wrapper by hand for models which I use.

[ostefano]

I might have another test case (feel free to create another issue if needed).

datamodel-codegen --url https://csrc.nist.gov/schema/nvd/api/2.0/cve_api_json_2.0.schema --input-file-type jsonschema --output ./cve_nvd/ --output-model-type pydantic_v2.BaseModel --target-python-version 3.11

The generated code has clear circular dependencies.

[pveierland]

Due to issues with circular references I wrote a small program to transform the schema source to a format that could be generated to a single Python file. There were still issues with type references that were not fixed with from __future__ import annotations, so I wrote the following hack that turns referenced generated names into forward declarations.

This fixes the issue for me currently - leaving it here as a hack in case it might help someone else. YMMV.

def transform_python_source_internal_forward_references(
    python_source: str,
) -> str:
    import ast

    class Parentage(ast.NodeTransformer):
        parent = None

        def visit(self, node):
            node.parent = self.parent
            self.parent = node

            node = super().visit(node)

            if isinstance(node, ast.AST):
                self.parent = node.parent

            return node

    class TransformForwardDeclarations(ast.NodeTransformer):
        def visit_Name(self, node):
            # NB: Does not transform names which are direct class bases,
            # or names which are used as attributes, e.g.: some_model.model_rebuild()
            replace_name_with_forward_declaration = (
                node.id in python_source_class_names
                and node.ctx
                and isinstance(node.ctx, ast.Load)
                and (
                    not node.parent
                    or not isinstance(node.parent, (ast.ClassDef, ast.Attribute))
                )
            )

            if replace_name_with_forward_declaration:
                return ast.Constant(value=node.id)
            else:
                return node

    python_source_ast = ast.parse(python_source)

    # Add parent information to individual nodes:
    Parentage().visit(python_source_ast)

    # Find class names defined in source:
    python_source_class_names = [
        node.name
        for node in ast.walk(python_source_ast)
        if isinstance(node, ast.ClassDef)
    ]

    # Replace references to class names with forward declarations:
    python_source_ast = ast.fix_missing_locations(
        TransformForwardDeclarations().visit(python_source_ast)
    )

    return ast.unparse(python_source_ast)