Suggestion for dynamic projections

[gethvi]

Hi,

I found a use case where I need to have projections made dynamically, basically just by specifying an arbitrary subset of fields that I want. My solution was to dynamically create new Model and pass it to the project() function. I was wondering if you would accept something like this as a PR? Would it be useful for Beanie? Or would you suggest another way of doing this? Thanks!

Right now it I am using it like this:

Product.find_many().project(create_submodel(Product, fields=["name", "description"]))

If made as part of Beanie, could it be like this?

Product.find_many().project(fields=["name", "description"])

PoC source with an example:

from typing import List, Union, Dict, Type, Optional
from pydantic import create_model, BaseModel


def create_submodel(
    model_type: Type[BaseModel], fields: List[Union[str, Dict]]
) -> Type[BaseModel]:
    """
    Creates a new model based on specified model_type with only a subset of it's fields.

    :param model_type:
    :param fields:
    :return:
    """
    model_fields = model_type.__fields__
    model_field_names = set(model_fields.keys())
    submodel_fields = {}
    for field in fields:

        if isinstance(field, str):

            if field not in model_field_names:
                raise ValueError(
                    f"field '{field}' not defined in model {model_type.__name__}"
                )

            submodel_fields[field] = (
                model_fields[field].type_,
                model_fields[field].field_info,
            )

        elif isinstance(field, dict):
            field.values()
            field_name = list(field.keys())[0]

            if field_name not in model_field_names:
                raise ValueError(
                    f"field '{field_name}' not defined in model {model_type.__name__}"
                )

            submodel = create_submodel(
                model_fields[field_name].type_, field[field_name]
            )
            submodel_fields[field_name] = (submodel, model_fields[field_name].field_info)

    return create_model(f"{model_type.__name__}SubModel", **submodel_fields)


if __name__ == "__main__":

    class Category(BaseModel):
        name: str
        description: str

    class Product(BaseModel):
        name: str
        description: Optional[str] = None
        price: float
        category: Category

    ProductSubModel = create_submodel(
        Product, fields=["name", "price", {"category": ["name"]}]
    )

    product_with_less_info = ProductSubModel.parse_obj(
        {"name": "Banana", "price": 3.14, "category": {"name": "Fruit"}}
    )

    print(product_with_less_info.json(indent=2))

# RESULT:
# {
#   "name": "Banana",
#   "price": 3.14,
#   "category": {
#     "name": "Fruit"
#   }
# }

[roman-right]

Hey @gethvi ,

This feature looks interesting. I think it would be nicer to have smth like model factory (as you have in your example) instead of providing the list of the fields to the project method. In this case, it will be simpler to add additional API like:

from typing import Union, Set, Dict, Any

from pydantic import BaseModel


def cut_model(
        name: str,
        model: BaseModel,
        include: Union[Set, Dict[str, Any]],
        exclude: Union[Set, Dict[str, Any]]
) -> BaseModel:
    ...

I have to think more about the implementation and interface, but the idea is really good. Thank you very much.

[anth2o]

Hello,

@gethvi I really like your solution, but it would also be interesting to handle use cases such as:

class B(BaseModel):
    z: int
    y: str = None

class A(BaseModel):
    x: str
    w: Dict[str, B]

C = create_submodel(A, fields=["x", {"w": {"foo": {"z"}}}])
results = A.find_many().project(projection_model=C)

then , results[0].dict() would look like:

{"x": "value_of_x", "w": {"foo": {"z": 1}}}

Your current implementation fails with:

    def create_submodel(
        model_type: Type[BaseModel], fields: List[Union[str, Dict]]
    ) -> Type[BaseModel]:
        """
        Creates a new model based on specified model_type with only a subset of it's fields.

        :param model_type:
        :param fields:
        :return:
        """
        model_fields = model_type.__fields__
        model_field_names = set(model_fields.keys())
        submodel_fields = {}
        for field in fields:

            if isinstance(field, str):

                if field not in model_field_names:
>                   raise ValueError(
                        f"field '{field}' not defined in model {model_type.__name__}"
                    )
E                   ValueError: field 'foo' not defined in model B

Do you think your solution could be adapted to fit this use case ? I didn't manage to do it by myself. Thanks

[lsaint]

#202 #204

i hope this suggestion can fix this 2 issuse too.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been stalled for 14 days with no activity.

[slingshotvfx]

This is a bit old, but I'm running into the same need today. I have a fairly complicated nested model I need to project arbitrary values from. In native mongodb syntax you can do stuff like:

projection = {Model.name: 1, Model.config.nested.data: 1, Model.config.nested.settings: 1}

So honestly I'd love if you could use Beanie's project() method with a native mongo dict like that.

Building up a Pydantic model with so many sub-models is quite clumsy to have to do by hand.