mypy

mypy copied to clipboard

(Not sure if it should be here or in typing)

Consider this function:

def f():
    return "hello"
def g(x):
    if x: return A(1)
    else: return A(2)

f obviously (without any deep analysis) returns str, and g returns A. Why not use this information? This pattern is very common, and taking advantage of it can help in precise checking and remove clutter (for example -> Union[Tuple[int, int, str], Tuple[int, str, bool]]).

I propose treating calls to functions whose return expressions consists solely of literals and constructor calls as if they were declared with the returned type (join or union of the return types).

Well, I always felt that the main reason for this was to allow mypy to be used on already-existing codebases, so the unannotated functions don't need to be type-safe (yet). This would change that behavior.

@kirbyfan64 I think I agree with what you are saying, but I also think there isn't a reason something like this couldn't be behind a flag, I've heard from several people that this is a significant pain point, as people don't want to have to do a lot of work to annotate obvious functions, so big +1 from me.

@kirbyfan64 the proposal does not require unannotated functions to be type-safe. It simply allows annotated functions that use them to be slightly more type safe (peek into the immediate return expressions, if there is one).

To elaborate:

class A: pass

def f(): return A(1)  # no error here, even though A has no no-arg __init__

def g() -> None:
    f().bar()  # error here

However when is it obvious enough? Where do you draw the line?

On Dec 25, 2017 13:43, "Elazar Gershuni" [email protected] wrote:

To elaborate:

class A: pass def f(): return A(1) # no error here, even though A has no no-arg init def g() -> None: f().bar() # error here

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/python/mypy/issues/4409#issuecomment-353889908, or mute the thread https://github.com/notifications/unsubscribe-auth/ACwrMlh2ba7M5ihwiKF6_8U6p7N_2f9Kks5tEAjhgaJpZM4RMQzD .

Either a literal or a direct constructor call on a globally defined class. (Part of the idea is that the return statements are also the documentation).

Expression atoms would be straightforward enough (not certain about identifiers however). At some point this could probably be expanded to entire expressions I believe (down the road).

Yes, that could be a nice path. Atoms, then (recursively) tuples/lists, then simple names. My hope is this could be done as part of semantic analysis, just like the analysis of the annotations themselves.

Could a director constructor call really be safe though? e.g.:

def func():
    A = lambda x: x
    return A(None)

Or even:

def func():
    return A(1)

func.__globals__['A'] = lambda x: x

Of course, this isn't ideal, but it happens in unannotated code.

I think it will be as safe as any other analysis performed by mypy. The first example is not a call to a global constructor; the second example defeats any other analysis.

I guess the discussion can be separated to two orthogonal questions:

• What should be considered as "obvious" return type
• How this mechanism is enabled: a flag for unannotated functions, a flag for partially-annotated functions, using an Infer annotation, or some other means

Note there is an old proposal https://github.com/python/typing/issues/276 to make this explicit. During the discussion it became clear that inferring types for arguments can be very hard, but for return types it is probably OK.

Recently, additional interest in this appeared in the context of data classes (PEP 557), where annotations are syntactically required, but people may be lazy to write exact types, so that this could be allowed:

@dataclass
class C:
    x: ... = 5  # equivalent to 'x = 5' (without an annotation) for a type checker

while for function this will work as

def func() -> ...:
    return 5

reveal_type(func())  # Revealed type is 'builtins.int'

An additional argument in favour of ellipsis is that libraries in numeric stack use ellipsis for inferred number of dimensions.

One problem with annotations is that they mean that the function will be checked. It also does not completely address the issue of clutter and laziness.

I agree that if this is the direction, then the ellipsis idea is nice. But two minor points should be noted: its meaning is unrelated to Tuple[int, ...], and ... everywhere might make the code look bad.

@elazarg

One problem with annotations is that they mean that the function will be checked.

It is maybe a matter of taste, but I think it is rather a plus, since I like to keep the current separation between annotated and non-annotated functions.

The idea is to not to make the human hands robotic, but to give them fingernails 😄

Behind a flag, the taste will be the user's.

The problem with a flag is that there already lots of them, so I am not keen to have one more (and this also doesn't solve the @dataclass problem). Anyway, I think we need to hear opinions of others.

I think this definitely should be behind a flag, if included, because it changes the spirit of mypy by not allowing typed functions. There is a similar discussion for typescript, although it's for function arguments rather than return type ( https://github.com/Microsoft/TypeScript/issues/15114).

@ethanhs who specifically said the lack of this feature was a pain point (just wondering)?

I agree with Guido. It seems that the dividing line will have to be kind of arbitrary.

Somewhat related: One potential longer-term workaround to the problem would be to provide an editor hook that could automatically create a signature for the current function being edited. It could infer simple enough types and leave placeholders for other types. Figuring out the types could be a feature of the mypy daemon mode. Basically this would integrate PyAnnotate-like functionality to an editor.

@elliott-beach I don't think this is related. I suggest no inference at all, let alone cross-procedural one.

Typescript does this by default and it works beautifully. They have an (encouraged) compiler flag 'noImplicitAny' which will cause an error if the return type cannot be inferred; in this case the dev should manually annotate. The end result is really nice - I get return type safety for free in most cases.

The question about "when is it obvious enough" is pushed under the carpet and answered with "when our static analyzer can understand it". The analyzer improves with most typescript releases, so things that used to require explicit annotation (remember, the user knows this because they will have 'noImplicitAny' turned on) may no longer need them in newer Typescript versions. This has not caused any problems that I am aware of.

It does clash with the current mypy use of type annotations to determine whether to check a function or not. To be honest I reckon this is a pain. I would rather just be able to tell mypy "check this whole file" or if really needed, just use a decorator on specific functions to opt in or out. Other languages (rust, new c++ features, new c# features, swift, ts) are all heavily embracing type inference everywhere, and as a dev I really find this direction to be a lot more ergonomic (safety with less typing [of the keyboard variety], what'd not to like?)

Maybe we can add a marker to request this, e.g. -> infer. But I presume that even TypeScript doesn't infer the argument types, so this feature seems of limited value except for argument-less functions.

To answer your question "what's not to like", it depends on whether you're working on a small new program or trying to add annotations to millions of lines of legacy code. In the latter case, turning on checking for all code would cause too many false positives (since legacy code tends to use clever hacks that the type checker doesn't understand). If you're in the small-new-program camp, just turn on --check-untyped-defs and other strict flags.

Maybe we can add a marker to request this, e.g. -> infer

and there is an old proposal for this https://github.com/python/typing/issues/276

Would it be weird to only do this when --check-untyped-defs is true? Then we can side-step the ambiguity of un-annotated functions and people can opt into this behavior instead of opt out (plus no new flag needed).

Would it be weird to only do this when --check-untyped-defs is true?

I like this idea!

I like this too.

I like this idea!

I like this too.

Great! Perhaps @elazarg is still interested? If not I might take a shot at it if I can find the time.

That is a good idea indeed.

I'm still interested (and haven't contributed anything for too long) but I cannot commit to it before mid-April.

I'm still interested (and haven't contributed anything for too long) but I cannot commit to it before mid-April.

Glad to hear you are still interested! I think this is definitely a "nice to have" feature so if you find time in the future, great, but no need to get it done soon :)

I'm removing the low priority and 'needs discussion' labels since this seems like a popular enough idea. We'd still need somebody to contribute an implementation.

I think that instead of special casing literals and constructor calls, this should use the normal type inference engine (perhaps with some limitations). Otherwise the behavior would be too ad hoc.

For example, these should work:

def f():
    return sys.platform.startswith('win')   # Infer bool as the return type

def g():
    return f()  # Infer bool as the return type

def h():
    return [g(), g()]  # Infer list[bool] as the return type

The type checker already has the concept of 'deferring' type checking of functions if we refer to something without a ready type. In cases like the above, the type of the function would not be ready until we've type checked the body of the function. After type checking we'd infer the return type from the return statements.

The implementation could slow down type inference a lot, so we may need to be more clever about the order of type checking. A minor performance hit would be fine, however.

Not sure if this is the best place to suggest this, or should I make a new issue, but I've recently tried using MyPy with the --disallow-incomplete-defs flag on my codebase, and the results I got were quite off-putting, after a bunch of functions that had typed input paremeters, but no -> None return type defined, were returned as problems. While I get the meaning behind the flag (and the results I got), there are cases where a function doesn't even have a return statement inside and isn't used in any expression either (only simple call) - some examples of those are __init__ or helper-setter methods on a class, for example. I find having to add -> None to every function like that quite pointless, as MyPy should be able to detect that easily.

My suggestion would be to infer the return type as None (something that seems to fit within this issue), if the body of the function contains a bare return statement, or no return statement at all. Right now, this is how it looks like:

# mypy: disallow-incomplete-defs

from typing import List

strings: List[str] = []

def add(text: str):  # error: Function is missing a return type annotation
    strings.append(text)