Explore better typing for comprehensions / computational expressions

[virusdave]

Consider:

    @effect.option
    def fn() -> Generator[Any, Any, List[str]]:
        x: int = yield 42
        y: str = yield f"{x} as a string"
        z: List[str] = yield from Some([y, str(x)])
        return z

Currently, unless each expression has the same type, you're stuck either being untyped or Any-typed and manually typing each bound name. This is really unfortunate and not at all type-safe.

Chaining a bunch of transformations of values within a functor or monad is a very common use of Haskell do-expressions, Scala for-expressions, and presumably (although i have no first hand knowledge) F# computational expressions.

It's really nice to be able to chain these and rely on types to ensure the correctness of each step.

With the current approach, it doesn't seem possible to type the individual layers of unwrapping differently unless they all share the same contained type (for the contravariant type parameter). This is really limiting the usefulness, alas.

Context: I'm trying to introduce some good foundations and abstractions for correctness at my company, and I think Expression could be a part of this based on its trajectory. However, the current limitations/ergonomics like this, combined with limitations in mypy would make this a somewhat difficult sell, so I'm hoping there's a better approach or some ideas for improvement. Happy to assist where possible, but i'm definitely not an expert in python or python typing.

[dbrattli]

Ok, I think I understand the problem. Not sure if this can be solved, but I will see if there anything that can be done to fix this or workarounds. The current problem is that we are sort of being an advanced list comprehension, and the list needs to have a given type T and every yield/send also needs to have a fixed type.

There is not any generic way (syntactic sugar) to achieve such unwrap / co-monadic / cata behavior in Python. In F# there is let! that unwraps and is basically a bind function behind the scenes. In Python there is yield from and await where the latter is intended for asynchronous workflows. There's also for ... in and with ... as that could be used and is currently being (mis)used by pattern matching in Expression. This could perhaps also be used to allow us to unwrap other types, but will increase the nesting for dependent computations e.g:

    @effect.option
    def fn() -> Generator[str, str, str]:
        z: str
        for x in Some(42.0):
            for y in Some(int(x)):
                z = yield from Some(str(y))

        return z

Nesting could actually be removed but we are seriously starting to misuse Python constructs 😬

    @effect.option
    def fn() -> Generator[str, str, str]:
        z: str
        for x in Some(42.0):
            ...
        for y in Some(int(x)):
            ...
        z = yield from Some(str(y))

        return z

Another way would be to (mis)use async / await e.g:

    @effect.option
    async def fn() -> str:
        x = await Some(42.0)
        y = await Some(int(x))
        z = await Some(str(y))

        return z

[bgrounds]

Hi. Very nice library :)

I found this issue while trying to get this function to type check without the explicit annotations.

This type checks, but the annotations are cumbersome:

@effect.option
def maybe_add(maybe_a: Option[int], maybe_b: Option[int]) -> Option[int]:
    a: int = yield from maybe_a
    b: int = yield from maybe_b
    return a + b

... this has no type annotations, but doesn't type check:

@effect.option
def maybe_add(maybe_a: Option[int], maybe_b: Option[int]) -> Option[int]:
    a = yield from maybe_a
    b = yield from maybe_b
    return a + b

... but while playing around, I found that this does type check:

@effect.option
def maybe_add(maybe_a: Option[int], maybe_b: Option[int]) -> Option[int]:
    yield from (a + b for a in maybe_a for b in maybe_b)

Nice! And if we have more than a one-liner, this works too:

@effect.option
def maybe_add(maybe_a: Option[int], maybe_b: Option[int]) -> Option[int]:
    yield from (
        a + b
        for a in maybe_a
        for b in maybe_b
    )

It's like a backward do-notation, which is close enough for me :)

I'm type checking with pyright 1.1.155

[dbrattli]

Wow, that' really nice! Thanks for sharing! I'll see if I can add that to the docs somewhere

[ssjw]

@bgrounds This type checks correctly without annotating the variables. Also, return type is correctly typed to Option[int] instead of Option[Any] (I replied about that in another issue).

@OptionBuilder[int]()
def maybe_add(maybe_a: Option[int], maybe_b: Option[int]) -> Generator[int, int, Option[int]]:
    a = yield from maybe_a
    b = yield from maybe_b
    return Some(a + b)