AttributeError while creating object with nested data/model

[kido5217]

First Check

• [X] I added a very descriptive title to this issue.
• [X] I used the GitHub search to find a similar issue and didn't find it.
• [X] I searched the SQLModel documentation, with the integrated search.
• [X] I already searched in Google "How to X in SQLModel" and didn't find any information.
• [X] I already read and followed all the tutorial in the docs and didn't find an answer.
• [X] I already checked if it is not related to SQLModel but to Pydantic.
• [X] I already checked if it is not related to SQLModel but to SQLAlchemy.

Commit to Help

• [x] I commit to help with one of those options 👆

Example Code

from typing import List, Optional

from sqlmodel import Field, Relationship, Session, SQLModel, create_engine

device_data = {
    "name": "Router",
    "interfaces": [
        {
            "name": "eth0",
            "description": "Ethernet 0",
        },
        {
            "name": "eth1",
            "description": "Ethernet 1",
        },
    ],
}


class Device(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)

    name: str = Field(index=True)

    interfaces: List["Interface"] = Relationship(back_populates="device")


class Interface(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)

    name: str = Field(index=True)
    description: str

    device_id: Optional[int] = Field(default=None, foreign_key="device.id")
    device: Optional[Device] = Relationship(back_populates="interfaces")


def main():

    engine = create_engine("sqlite:///database.db", echo=True)
    SQLModel.metadata.create_all(engine)

    with Session(engine) as session:
        device = Device.parse_obj(device_data)
        session.add(device)
        session.commit()


if __name__ == "__main__":
    main()

Description

I'm trying to use nested models to parse structured data and store it in DB. In test program, example dataset is created in variable device_data

After creating test data and Device and Interface models with One-to-Many relationship, I'm trying to create device object: device = Device.parse_obj(device_data), but I'm getting exception: AttributeError: 'dict' object has no attribute '_sa_instance_state'

Operating System

Linux

Operating System Details

No response

SQLModel Version

0.0.6

Python Version

Python 3.10.4

Additional Context

Full output:

2022-04-03 01:17:34,890 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2022-04-03 01:17:34,890 INFO sqlalchemy.engine.Engine PRAGMA main.table_info("device")
2022-04-03 01:17:34,890 INFO sqlalchemy.engine.Engine [raw sql] ()
2022-04-03 01:17:34,892 INFO sqlalchemy.engine.Engine PRAGMA temp.table_info("device")
2022-04-03 01:17:34,892 INFO sqlalchemy.engine.Engine [raw sql] ()
2022-04-03 01:17:34,892 INFO sqlalchemy.engine.Engine PRAGMA main.table_info("interface")
2022-04-03 01:17:34,892 INFO sqlalchemy.engine.Engine [raw sql] ()
2022-04-03 01:17:34,892 INFO sqlalchemy.engine.Engine PRAGMA temp.table_info("interface")
2022-04-03 01:17:34,893 INFO sqlalchemy.engine.Engine [raw sql] ()
2022-04-03 01:17:34,893 INFO sqlalchemy.engine.Engine 
CREATE TABLE device (
        id INTEGER, 
        name VARCHAR NOT NULL, 
        PRIMARY KEY (id)
)


2022-04-03 01:17:34,893 INFO sqlalchemy.engine.Engine [no key 0.00006s] ()
2022-04-03 01:17:34,901 INFO sqlalchemy.engine.Engine CREATE INDEX ix_device_name ON device (name)
2022-04-03 01:17:34,901 INFO sqlalchemy.engine.Engine [no key 0.00013s] ()
2022-04-03 01:17:34,912 INFO sqlalchemy.engine.Engine 
CREATE TABLE interface (
        id INTEGER, 
        name VARCHAR NOT NULL, 
        description VARCHAR NOT NULL, 
        device_id INTEGER, 
        PRIMARY KEY (id), 
        FOREIGN KEY(device_id) REFERENCES device (id)
)


2022-04-03 01:17:34,912 INFO sqlalchemy.engine.Engine [no key 0.00020s] ()
2022-04-03 01:17:34,921 INFO sqlalchemy.engine.Engine CREATE INDEX ix_interface_name ON interface (name)
2022-04-03 01:17:34,921 INFO sqlalchemy.engine.Engine [no key 0.00014s] ()
2022-04-03 01:17:34,930 INFO sqlalchemy.engine.Engine COMMIT
Traceback (most recent call last):
  File "/home/kido/projects/test/python/pydser/./pyser.py", line 52, in <module>
    main()
  File "/home/kido/projects/test/python/pydser/./pyser.py", line 46, in main
    device = Device.parse_obj(device_data)
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlmodel/main.py", line 578, in parse_obj
    return super().parse_obj(obj)
  File "pydantic/main.py", line 511, in pydantic.main.BaseModel.parse_obj
  File "<string>", line 4, in __init__
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/state.py", line 479, in _initialize_instance
    with util.safe_reraise():
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/state.py", line 477, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlmodel/main.py", line 518, in __init__
    setattr(__pydantic_self__, key, data[key])
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlmodel/main.py", line 528, in __setattr__
    set_attribute(self, name, value)
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 2255, in set_attribute
    state.manager[key].impl.set(state, dict_, value, initiator)
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1602, in set
    collections.bulk_replace(
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/collections.py", line 843, in bulk_replace
    appender(member, _sa_initiator=initiator)
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/collections.py", line 1169, in append
    item = __set(self, item, _sa_initiator)
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/collections.py", line 1134, in __set
    item = executor.fire_append_event(item, _sa_initiator)
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/collections.py", line 753, in fire_append_event
    return self.attr.fire_append_event(
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1429, in fire_append_event
    value = fn(state, value, initiator or self._append_token)
  File "/home/kido/projects/test/python/pydser/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1765, in emit_backref_from_collection_append_event
    child_state, child_dict = instance_state(child), instance_dict(child)
AttributeError: 'dict' object has no attribute '_sa_instance_state'

[kido5217]

I've managed to make it work with root_validator that finds all relationship and manually converts them to subclasses:

class Device(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)

    name: str = Field(index=True)

    interfaces: List["Interface"] = Relationship(back_populates="device")

    @root_validator(pre=True)
    def init_sublasses(cls, values):
        for relationship in cls.__sqlmodel_relationships__.keys():
            subclass = globals()[
                cls.__annotations__[relationship].__args__[0].__forward_arg__
            ]
            if isinstance(values.get(relationship), list):
                objs = []
                for interface_data in values[relationship]:
                    objs.append(subclass.parse_obj(interface_data))
                values[relationship] = objs
        return values

But I'm not sure if it's the proper way. It looks kind of hacky.

[samstiyer]

Would love to know if there is a more effective solution than what is posted above?

[dannyrohde]

I've done a similar workaround. You might want to make your code a bit more robust to deal with relationships that are not lists, but otherwise mine looks similar.

In general, I've found a couple of traps, where I used some Pydantic functionality just to find that it doesn't quite translate into SQLModel (yet).

[damarvin]

I stumbled in the same issue. I've no idea how, but as this is thrown in a stack depth of 13 in SQLAlchemy in my case, I vote for a more helpful error message, maybe catch the situation earlier.

[hugocool]

I am also facing the same issue in trying to model the response of an AWS Cli response:

from typing import Dict, List, Optional

from sqlmodel import JSON, Column, Field, Relationship, SQLModel

class Container(SQLModel, table=True):
    image: str = Field(primary_key=True)
    vcpus: int
    memory: int
    command: List[str]
    executionRoleArn: str
    volumes: List[str]
    environment: List[Dict] = Field(sa_column=Column(JSON))
    mountPoints: List[Dict] = Field(sa_column=Column(JSON))
    ulimits: List[Dict] = Field(sa_column=Column(JSON))
    networkInterfaces: List[Dict] = Field(sa_column=Column(JSON))
    resourceRequirements: List[Dict] = Field(sa_column=Column(JSON))
    secrets: List[Dict] = Field(sa_column=Column(JSON))
    exitCode: Optional[int]
    containerInstanceArn: Optional[str]
    taskArn: Optional[str]
    logStreamName: Optional[str]
    networkInterfaces: List[Dict] = Field(sa_column=Column(JSON))
    jobs: List["Job"] = Relationship(back_populates="container")

    class Config:
        arbitrary_types_allowed = True


class Job(SQLModel, table=True):
    jobArn: str
    jobName: str
    jobId: str = Field(primary_key=True)
    jobQueue: str
    status: str
    attempts: List[Dict] = Field(sa_column=Column(JSON))
    statusReason: Optional[str]
    createdAt: int
    startedAt: Optional[int]
    stoppedAt: Optional[int]
    dependsOn: List[Dict] = Field(
        sa_column=Column(JSON)
    )  # List[Optional["Job"]] = Relationship(back_populates="dependents")
    jobDefinition: str
    parameters: Dict = Field(sa_column=Column(JSON))
    timeout: Dict = Field(sa_column=Column(JSON))
    tags: Dict = Field(sa_column=Column(JSON))
    platformCapabilities: List[str] = Field(sa_column=Column(JSON))
    eksAttempts: List[Dict] = Field(sa_column=Column(JSON))
    container: Optional[Container] = Relationship(back_populates="jobs")
    container_id: Optional[str] = Field(default=None, foreign_key="container.image")

    class Config:
        arbitrary_types_allowed = True

Now when I try to parse a response:

from pydantic import parse_obj_as
from typing import List
import boto3

client = boto3.client("batch")

jobs_resp = client.describe_jobs(jobs=[
		...
    ])["jobs"]

jobs = parse_obj_as(List[Job], jobs_resp)

I get the following error:

.venv/lib/python3.9/site-packages/sqlmodel/main.py in validate(cls, value)
    590             if validation_error:
    591                 raise validation_error
--> 592             model = cls(**value)
    593             # Reset fields set, this would have been done in Pydantic in __init__
    594             object.__setattr__(model, "__fields_set__", fields_set)

 in __init__(__pydantic_self__, **data)

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlalchemy/orm/state.py in _initialize_instance(*mixed, **kwargs)
    480         except:
    481             with util.safe_reraise():
--> 482                 manager.dispatch.init_failure(self, args, kwargs)
    483 
    484     def get_history(self, key, passive):

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py in __exit__(self, type_, value, traceback)
     68             self._exc_info = None  # remove potential circular references
     69             if not self.warn_only:
---> 70                 compat.raise_(
     71                     exc_value,
     72                     with_traceback=exc_tb,

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py in raise_(***failed resolving arguments***)
    206 
    207         try:
--> 208             raise exception
    209         finally:
    210             # credit to

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlalchemy/orm/state.py in _initialize_instance(*mixed, **kwargs)
    477 
    478         try:
--> 479             return manager.original_init(*mixed[1:], **kwargs)
    480         except:
    481             with util.safe_reraise():

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlmodel/main.py in __init__(__pydantic_self__, **data)
    514         for key in non_pydantic_keys:
    515             if key in __pydantic_self__.__sqlmodel_relationships__:
--> 516                 setattr(__pydantic_self__, key, data[key])
    517 
    518     def __setattr__(self, name: str, value: Any) -> None:

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlmodel/main.py in __setattr__(self, name, value)
    523             # Set in SQLAlchemy, before Pydantic to trigger events and updates
    524             if getattr(self.__config__, "table", False) and is_instrumented(self, name):
--> 525                 set_attribute(self, name, value)
    526             # Set in Pydantic model to trigger possible validation changes, only for
    527             # non relationship values

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py in set_attribute(instance, key, value, initiator)
   2254     """
   2255     state, dict_ = instance_state(instance), instance_dict(instance)
-> 2256     state.manager[key].impl.set(state, dict_, value, initiator)
   2257 
   2258 

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py in set(self, state, dict_, value, initiator, passive, check_old, pop)
   1267                 )
   1268 
-> 1269         value = self.fire_replace_event(state, dict_, value, old, initiator)
   1270         dict_[self.key] = value
   1271 

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py in fire_replace_event(self, state, dict_, value, previous, initiator)
   1293 
   1294         for fn in self.dispatch.set:
-> 1295             value = fn(
   1296                 state, value, previous, initiator or self._replace_token
   1297             )

~/ISCO-Classification/.venv/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py in emit_backref_from_scalar_set_event(state, child, oldchild, initiator)
   1728         if child is not None:
   1729             child_state, child_dict = (
-> 1730                 instance_state(child),
   1731                 instance_dict(child),
   1732             )

AttributeError: 'dict' object has no attribute '_sa_instance_state'

From trial and error I nailed it down to the relationship between Job and Container, (basically I commented out all the fields and added them back in until the errors arose).

I adjusted the aforementioned script as follows:

    @root_validator(pre=True)
    def init_sublasses(cls, values):
        for relationship in cls.__sqlmodel_relationships__.keys():
            subclass = globals()[cls.__annotations__[relationship].__args__[0].__name__]
            if isinstance(values.get(relationship), list):
                objs = []
                for v in values[relationship]:
                    objs.append(subclass.parse_obj(v))
                values[relationship] = objs
            elif isinstance(values.get(relationship), dict):
                obj = subclass.parse_obj(values[relationship])
                values[relationship] = obj
        return values

And now it works for my case.

Could we integrate this into the SQLMODEL library?