Exclude any FK fields in pydantic_model_creator

[AntonZN]

class User(models.Model):
    id = fields.IntField(pk=True)
    username = fields.CharField(max_length=128, index=True, unique=True)
    email = fields.CharField(max_length=128, unique=True, index=True)

class Event(models.Model):
    id = fields.IntField(pk=True)
    name = fields.CharField(max_length=128)
    user = fields.ForeignKeyField("models.User", related_name="events")

It would be nice to add such a solution:

Event_Pydantic = pydantic_model_creator(Event, exclude=("user__email",))

[dstlny]

Why not just use “PydanticMeta” on your User model?

PydanticMeta

This would look something like:

class User(models.Model):
    id = fields.IntField(pk=True)
    username = fields.CharField(max_length=128, index=True, unique=True)
    email = fields.CharField(max_length=128, unique=True, index=True)

    class PydanticMeta:
        exclude = (“email”, )

pydantic_model_creator will then automatically pick these up :)

[AntonZN]

Let's say we have more model fields

class User(models.Model):
    id
    username
    email
    is_staff
    is_superuser
    user_permissions
    
     class PydanticMeta:
        exclude = ("email", "is_staff", "is_superuser", "user_permissions" )

Suppose we have a method in which I want to return a list of events and in this list I do not need additional information about the user, except for the id and username. If I use your way of processing, this will apply to all subsequent methods. In the / events / method I will get the desired data, and in the / profile / method I need detailed information about the user.

Such a solution will give more flexible pydantic models:

User_Pydantic = pydantic_model_creator(User)
Event_List_Pydantic = pydantic_model_creator(Event, exclude=("user__email", "user__is_staff", "user__is_superuser",))
Event_Single_Pydantic = pydantic_model_creator(Event)

[marcoaaguiar]

You can already do that, but you need to use '.' as separators:

Event_List_Pydantic = pydantic_model_creator(Event, exclude=("user.email", "user.is_staff", "user.is_superuser",))

[diegocgaona]

@marcoaaguiar I'm trying to do like you said related.fieldtobeexcluded

This is working? There is some documentation about it?

Best regards!

[w-A-L-L-e]

It works for me when I use exclude ['user.password'] however on other models/relations it does not seem to work. For instance I can do exclude['user.password', 'user.id', 'profile'] and then the related user.password and id is excluded and also the related profile. However when I do this exclude['user.password', 'profile.id'] the related profile.id is not excluded (which does work when doing it on a user.id) is this somehow only working on a 'user' relation?

BatchOutSchema = pydantic_model_creator(
    Batches, name="Batch",
    exclude=[
        'user.role',                 # works
        'user.password',        # works
        'user.created_at',      # works
        # 'user.id',                    # works
        # 'profile',                    # works
        # 'profile.id',                                   # does not work !!!
        # 'content_partner.created_at',   # does not work, again excluding content_partner entirely does work.
    ]
)

Also I haven't logged the queries yet. But does the exclude 'user' or 'profile' also make it so that the corresponding join is not done in the sql queries?

The strong thing is that '. operator' works fine with exclude for the Users model but not for BatchProfiles or ContentPartners. From the Batches model standpoint they are very similarly defined as ForeignKeyField fields:

class Batches(models.Model):
    id = fields.IntField(pk=True)
    title = fields.CharField(max_length=225)
    description = fields.TextField()
    content_partner = fields.ForeignKeyField("models.ContentPartners", related_name="batch")
    profile = fields.ForeignKeyField("models.BatchProfiles", related_name="batch")
    user = fields.ForeignKeyField("models.Users", related_name="batch")

class BatchProfiles(models.Model):
    id = fields.IntField(pk=True)
    name = fields.CharField(max_length=225)
  
class ContentPartners(models.Model):
    id = fields.IntField(pk=True)
    name = fields.CharField(max_length=225)
    created_at = fields.DatetimeField(auto_now_add=True)
    modified_at = fields.DatetimeField(auto_now=True)

class Users(models.Model):
    id = fields.IntField(pk=True)
    username = fields.CharField(max_length=20, unique=True)
    full_name = fields.CharField(max_length=50, null=True)
    password = fields.CharField(max_length=128, null=True)
    role = fields.ForeignKeyField("models.UserRoles", related_name="user", null=True)
    created_at = fields.DatetimeField(auto_now_add=True)
    modified_at = fields.DatetimeField(auto_now=True)

Update so I logged the queries and the good news is that indeed the exclude does indeed make the select for that relation is not done anymore. For instance excluding 'content_partner' makes following select disappear in sql logs:

SELECT "created_at","id","modified_at","name" FROM "contentpartners" WHERE "id" IN (1,1,1): None

Still no clue why doing exclude content_partner.created_at does not work however (it should just omit that in the select but it does not).

[w-A-L-L-e]

Is there a bugfix on the way somewhere for this or an alternative workaround? Today I have a similar issue where it's not possible to hide a specific column. In this case it was a computed column. But it always seems to happen with nested models (computed or not).

For instance Person model has many Photos:

class Person(Model):
    id = fields.IntField(pk=True)
    name = fields.CharField(max_length=255)
 
class Photo(Model):
    id = fields.IntField(pk=True)
    person = fields.ForeignKeyField("models.Person", related_name="photos")
    s3_object_name = fields.CharField(max_length=225, null=True)
    s3_thumbnail = fields.CharField(max_length=225, null=True)

    def s3_thumbnail_url(self) -> str:
        name, extension = os.path.splitext(self.s3_object_name)
        s3_file = f'{name}_thumbnail{extension}'
        s3_url = generate_s3_url(s3_file, expire_seconds=7200)
        print("signing s3 thumbnail: ", s3_url)
        return s3_url

    class Meta:
        ordering = ["id"]

    class PydanticMeta:
        computed = ['s3_thumbnail_url']
        exclude = ['s3_thumbnail']

So in photos we get back an s3_thumbnail_url (which is computed and a bit expensive) that we want to omit in the parent table people. The Photo pydantic serialization works fine. But I cant hide that s3_thumbnail_url column on person.

PeopleListSchema = pydantic_model_creator(
    Person, name="PeopleList", 
    exclude=[
        'photos.s3_thumbnail_url',      # does not work, the s3_thumbnail_url is still computed and given back  (is what I need here)
        'photos',                                     # does work, but is not what I need here, I still need the other data from person.photos
        'photo__s3_thumbnail_url',    # does not work
        'photos__s3_thumbnail_url',  # does not work
        'photo.s3_thumbnail_url',      # does not work
    ]
)

I've also tried using a separate meta_override. But it just doesn't want to work whenever when we try to hide a column on such related table (it works fine for a 1-1 relation or 1-n in the other direction, but not like we have here where 1 person has many photos and we need to omit a column on each of the photos associated with the person).

class PersonExportMeta:
    exclude = [
        'photos.s3_thumbnail_url', # same issue, this is just ignored by tortoise and 
    ]
        

PeopleExportSchema = pydantic_model_creator(
    Person,
    name="PeopleListExport",
    meta_override=PersonExportMeta
)

Any help or alternative approach on how to achieve this would be appreciated

[abondar]

@w-A-L-L-e I can probably look at it, but honestly it's quite hard to debug pydantic issues after migration to 2.0, because it is all rust code just linked to python

It would help if you will be able to create reproducible snippet of code (in style of examples in this repository), but I cannot promise I will be able to look into it fast, as I am quite busy lately