[BUG] Pydantic type transformer (msgpack) can't deal with nan numbers

[fg91]

Describe the bug

from flytekit.extras.pydantic_transformer.transformer import PydanticTransformer
from flytekit.core.context_manager import FlyteContext
from pydantic import BaseModel

ctx = FlyteContext.current_context()
tfm = PydanticTransformer()

class TestModel(BaseModel):
    a: float

literal = tfm.to_literal(ctx, TestModel(a=float('nan')), TestModel, tfm.get_literal_type(TestModel))

print(literal)

reconstructed = tfm.to_python_value(ctx, literal, TestModel)

print(reconstructed, type(reconstructed))

ValidationError: 1 validation error for TestModel
a
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/float_type

Expected behavior

I would have expected that the de-serialzed base model like the original contains a float('nan').

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• [x] Yes

Have you read the Code of Conduct?

• [x] Yes

[eapolinario]

Good catch.

First a bit of context. In the original flytekit PR we encountered a problem with supporting enums in pydantic models, so we decided to use pydantic's model_dump_json to get a json representation of the pydantic model and only then encode that using msgpack. Unfortunately, pydantic's model_dump_json doesn't handle NaN well by default (it turns it into a null), however if you modify the definition of the model slightly then this works:

from pydantic import BaseModel, ConfigDict
...
class TestModel(BaseModel):
    a: float

    model_config = ConfigDict(ser_json_inf_nan="constants")

[fg91]

Thanks for the hint @eapolinario 🙏

[github-actions[bot]]

"Hello 👋, this issue has been inactive for over 90 days. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏"

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 90 days and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏