[Query] Pydantic Validation throws error even while fetching data from db

[sushilkjaiswar]

I have added Pydantic native validation to the field but it triggers while reading from db. Is this functionality supported by beanie and how? Or we should avoid using validator functionality of Pydantic library along with beanie module.

class UserStore(CommonProperties):
    id:str = Field(default_factory=Helper.get_uuid_hex, alias="_id")
    first_name:str
    last_name:str
    username: Indexed(EmailStr, unique=True)
    password: str

    @validator('first_name')
    def id_must_be_4_digits(cls, v):
        if ' ' not in v :
            raise ValueError('must be 4 digits')
        return v

#  Below line throws validation error 
user = await UserStore.find_one(UserStore.username == username)

# output
# 1 validation error for UserStore
# first_name
     must be 4 digits **(type=value_error)**

[roman-right]

Hey @sushilkjaiswar , If data in the database doesn't fit the validator, it will raise an error, as beanie converts dicts, that came from the engine to pydantic model objects. And it uses pydantic tools for this.

[sushilkjaiswar]

I am facing issue during validating password with confirm password field. After plain text is matched I save encrypted password in the database. While fetching db and using this model it throws error because it will go through this validation every time which I should be able to avoid. I must avoid validation mostly during reading operation but since Pydantic's nature is to validate data it is going to evaluate every time we set data.

How to do you see this issue to solve using Pydantic or Beanie. I have implemented the crude way for now where validation operation happens outside Pydantic functionality.

class UserStore(CommonProperties):
      id:str = Field(default_factory=Helper.get_uuid_hex, alias="_id")
      first_name:str
      last_name:str
      username: Indexed(EmailStr, unique=True)
      password: str
      cpassword: str


      @validator('cpassword')
      def passwords_match(cls, v, values, **kwargs):
         encrypted = Helper.get_hashed_string(values['password'])
         if 'cpassword' in values and v != values['password']:
            raise ValueError('passwords do not match')
         values['password'] = encrypted
         return encrypted

[tataraba]

Hi @sushilkjaiswar,

Why would you need to have the cpassword field included in the model to begin with? Presuming that a user is sending you their password and cpassword, it makes more sense to check the match outside of the database scope anyway.

For example, I have a signup route on my website where a user enters email, and I ask for a password and repeat_password.

I can check that the password and repeat_password strings match. If not, I raise an exception. If so, only then do I go ahead and create a user in my database (or any other function that requires the password match.

This ensures that the validation is kept outside of your database models, and, as you point out, you don't need the validation to happen every time an operation happens on your database. The "signup" function might look a little something like:

async def signup(
    email: str,
    password: str,
    repeat_password: str
):

    user = await get_user(email)

    if user:
        raise HTTPException(status_code=400, detail="User already exists")

    if password != repeat_password:
        raise HTTPException(status_code=400, detail="Passwords do not match")

    await create_user(email, password)

note: the get_user and creat_user functions are basic crud operations you can define elsewhere.

Now, you can use your existing model without having to validate the matching password. You can just create a method in your class to take care of the password hashing automatically, but you can control when this happens.

Also, I'd advise not storing the password string in your database. You likely should encrypt and decrypt any time you want to retrieve the password. In that case, your model might look something like:

class UserStore(CommonProperties):
    id:str = Field(default_factory=Helper.get_uuid_hex, alias="_id")
    first_name:str
    last_name:str
    username: Indexed(EmailStr, unique=True)
    password: str
    hashed_password: str


    def hash_password(self, password):
        self.hashed_password = hash_password(password)

    def verify_password(self, password):
        return verify_password(password, self.hashed_password)

Hope that helps.

[sushilkjaiswar]

Hi @tataraba,

Apologies for late reply. I was away from work for few months.

Regarding your suggestion of validating password and cpassword out side database model, which I am already following.

Since beanie is a ODM which can do much more than database model handling. It was a suggestion to add feature in beanie for such kind of a feature validation, it will be great if validation is handled at ODM then we have our code more organised and clear separation of functionality/logic. Just imagine how easy and clear code will be if beanie ODM handles these types of validation which takes care inherently scenarios like this. I don't have to worry about having validation at two different places but in my ODM model itself and decides which properties must be stored in final record of the db and which should be omitted.

Since beanie is using pydantic where validation rules our inherited, I was curious if we can bring our own validation rules as well to handle other validation scenarios like password and confirm password and save only one field.

Hope this clears the thought which I have and would be interested to understand, if we can implement in beanie project with ease.

[roman-right]

Hi, @sushilkjaiswar , This sounds reasonable. I will think about depending on source validations. Like if the data is coming from db - one scenario, if from outside - another scenario. Using this it would be possible to add custom data types (and keep the most often used in Beanie) Thank you for this request

[sushilkjaiswar]

Hi @roman-right,

I was trying to achieve the validation using pydantic and below is the code,

from pydantic import BaseModel, ValidationError, validator
class UserModel(BaseModel):
    name: str
    username: str
    password1: str
    password: str

    @validator('name')
    def name_must_contain_space(cls, v):
        if ' ' not in v:
            raise ValueError('must contain a space')
        return v.title()

    @validator('password')
    def passwords_match(cls, v, values, **kwargs):
        if 'password1' in values and v != values['password1']:
            raise ValueError('passwords do not match')
        # print(values)
        del values['password1']
        return v

    @validator('username')
    def username_alphanumeric(cls, v):
        assert v.isalnum(), 'must be alphanumeric'
        return v


try:
    # Passess test when passwords are same 
    user_with_passwords_matched = UserModel(
        name='samuel colvin',
        username='scolvin',
        password1='zxcvbn',
        password='zxcvbn',
    )
    
    # TODO: check this output where I delete the property after validation is successfull. This is kind of a hack and is not a clean solution, we need to follow order of properties if this type of validation needs to work. 
    print("user_with_passwords_matched","\n", user_with_passwords_matched)
    
    # Passess the test when passwords are different and raises errors
    user_without_passwords_matched = UserModel(
        name='samuel sda',
        username='scolvin',
        password1='zxcvbn',
        password='zxcvbn2',
        
    )
    print("\n"*2)
    print("user_without_passwords_matched", "\n", user_without_passwords_matched)

except ValidationError as e:
    print("\n"*2,"user_without_passwords_matched","\n")
    print(e)
    """
    2 validation errors for UserModel
    name
      must contain a space (type=value_error)
    password2
      passwords do not match (type=value_error)
    """

I have posted above code example to have a look and see if you can get a better idea to solve this issue by using existing Pydantic validation feature. I would prefer to have decorator which applies validation rules and necessary flag to tell model whether field must be stored in the database or it should not.

As a suggestion, I would like you to go through SQLAlchemy ORM(if you have not already) which has lots of features and can be a motivation for bringing features to Beanie project, if you plan to see beanie growing as a potential ODM module.

Let me know if you need any help.

Thanks

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been stalled for 14 days with no activity.