Keyword args not supported

[DonkeyKong1]

First, I really like this. I was getting pretty confused when I kept getting "plum.function.NotFoundLookupError" and couldn't find issue with the simple class I'd made. Randomly, I forgot to provide a keyword for the arg and it worked. Am I correct that keyword args are not supported?

[wesselb]

Hey @DonkeyKong1! You're right that dispatch interacts a bit strangely with keyword arguments. In particular, arguments which are not given a default value must also be given in non-keyword form. E.g., consider

@dispatch
def f(x: int):
     return x

Then f(2) will work, whereas f(x=2) breaks.

You can use default values for arguments, and, in that case, you can give these arguments in keyword form. See https://github.com/wesselb/plum#keyword-arguments-and-default-values.

I hope this clarifies! :) Please let me know if you have any other questions.

[astanziola]

Hi! Sorry to post on an old issue, but is there any fundamental reason for not supporting keyword arguments without default values? I'm using plum to dispatch over operators with many type combinations of mandatory inputs, and it would make the code much clearer to be able to call them using keyword arguments.

[wesselb]

Hey @astanziola!

The design of Plum is heavily inspired by how multiple dispatch works in the Julia programming language. In Julia, keyword arguments are also not considered in the dispatch process. I believe you can find some discussion online around this point.

More fundamentally, multiple dispatch is based on the idea that arguments are identified by position and type. Keyword arguments, on the other hand, are identified by name. I agree that one could consider a dispatch process which also considers keyword arguments, but, up to this point, we've tried to follow the Julia programming language.

[astanziola]

Thank you for clarifying @wesselb , I had no idea that Julia was following this rule for the multiple dispatch system!

[teffland]

Hi, are there any plans in the future to support this? I really like dispatch, but this is a bit of an issue for downstream usability, since in python its very common to call functions with named args for both readability and to stay robust to irrelevant upstream signature updates.

Is there something inherently preventing this behavior? Would be interested in making a contribution to allow this behavior or at least display a more useful error message to the user.

[wesselb]

@teffland Currently, there are no plans to dispatch on keyword arguments, but obviously future plans can change. :) I agree that calling a function with named arguments is very common in Python. It would be very nice to support this.

Is there something inherently preventing this behavior?

That's a good question, and I'm inclined to answer "not necessarily", but how this would work would have to be spelled out in detail. For example, consider the following scenario:

@dispatch
def f(x: int, y: float):
    ... # Method 1

@dispatch
def f(y: float, x: int):
   ... # Method 2

Then calling f(1, 1.0) would call method 1 and calling f(1.0, 1) would call method 2.

What's unclear to me is what f(x=1.0, y=1) would do. For method 1, I suppose that the call would be equivalent to f(1.0, 1), which wouldn't match. For method 2, the call would be equivalent to f(1, 1.0), which also wouldn't match. The strange thing is that the arguments are switched around between the two methods because the names don't line up.

Perhaps this a poor example, but what's going wrong is the following. In Python, there are two ways to say that a particular argument should have a value:

1. by position, or
2. by name.

Once you name things, the position becomes irrelevant. And once you position an argument, the name of the argument as written in the function doesn't matter. Hence, in some sense, it's an either-or situation where you have to choose to designate arguments by position or designate arguments by name.

Currently, the whole multiple dispatch system has been designed around arguments-designated-by-position (and type), which is, as argued above, somehow incongruent with designating arguments by name. Why? Once you name things, the position becomes irrelevant; and once you position something, the name of the argument becomes irrelevant.

Therefore, if we were to support naming arguments, how precisely this would work would have to be spelled out in detail.

[teffland]

@wesselb thanks for the detailed response. I've given it some consideration.

I suppose one way to deal with this could be to take each function definition and convert it into multiple possible signatures. These signatures could be composed of one ordered and one named component, for the positional and named parts respectively. As in, Signature = Tuple[Tuple[Type],Dict[str,Type]].

Since python requires positional args come before named ones, a function with n parameters could be converted into n+1 total signatures, each with a different partition of arguments in the ordered vs named subsets. The 0th would have 0 elements specified by position and all in the named set, the 1st would be the first element specified by position and the rest in the named set, etc. As follows:

def f(x:int, y: int, z: int):
   # 4 possible signatures
   # 0. ((,),{x:int, y:int, z:int})
   # 1. ((int,), {y:int, z:int})
   # 2. ((int,int), {z:int})
   # 3. ((int,int,int),{})

Then when, matching signatures it would partition the call signature the same way.

One other smaller detail to consider would be matching named args with defaults vs w/o defaults. I guess for the above signatures, you could only put params in the "named" signature if they don't have defaults, and just would check that the input named arguments are a superset of the "named" portion of the signature. That would take care of it I believe. So, continuing the above example, if f were instead written as:

def f(x:int, y: int, z: int, a: int = 0):
   # 4 possible signatures
   # 0. ((,),{x:int, y:int, z:int})
   # 1. ((int,), {y:int, z:int})
   # 2. ((int,int), {z:int})
   # 3. ((int,int,int),{})

then it would still have the same signature. (Note that this would mean this definition is incompatible with the first definition in the same registry. I would argue this is a good restriction, as allowing two implementations of functions that only differ by optional parameters is inherently ambiguous.)

Let me know what you think. Would be happy to take a crack at a PR sometime.

[wesselb]

Hey @teffland! Apologies for the very slow reply.

What you're proposal is in fact already happening, but only for arguments with defaults. That is,

def f(x: int, y: int = 1, z: int = 2):
   # Generates three signatures:
   # 1. ((int,), {y: int = 1, z: int = 2})
   # 2. ((int, int), {z: int = 2})
   # 3. ((int, int, int), {})

Extending it to also include all position arguments is an interesting proposal! I might see potential issues. Consider the definitions

@dispatch
def f():
    print("Doing something with no ints")

@dispatch
def f(x: int):
    print("Doing something with one int")

@dispatch
def f(x: int, y: int):
    print("Doing something with two ints")

Then the third one would generate the methods ((), {x: int, y: int}) and ((int,), {y: int}), overwriting the definitions of the first two! What are your thoughts on potential conflicts like this?

[gabrieldemarmiesse]

I played a bit in plume and I would like to provide a small improvement: When the signature of all overloads are matching (same number of arguments and in the same order) then using keyword argument shouldn't be ambiguous.

I feel this use case is quite common and even if it's not all use cases of plum, it would remove some astonishment that can happen as a first time user of plum. It's quite confusing for beginners right now.

If the signature of all overloads aren't matching (different arguments or a different order) then we can fall back to the existing behaviour and fail.

[wesselb]

@gabrieldemarmiesse could you give an example? I'm not sure that I fully understand this use case.

[gabrieldemarmiesse]

When I'm talking about a function where all overloads have matching signatures are matching, I'm thinking about this:

from plum import dispatch

@dispatch
def foo(x: int, y: int):
    print("you have called in int version of foo")

@dispatch
def foo(x: str, y: str):
    print("you have called in str version of foo")

foo(1, 2)
foo("1", "2")
# this should be  possible but currently fail
# with plum.resolver.NotFoundLookupError
foo(x=1, y=2)

Interestingly, this type of restriction (same number of arguments and always in the same order) is also the restriction that is enforced by @overload, so I believe it's very common.

This common pattern should be supported, while this pattern:

@dispatch
def f(x: int, y: float):
    ... # Method 1

@dispatch
def f(y: float, x: int):
   ... # Method 2
f(x=1, y=2)

should fail with a discriptive error message saying that if you want to use keyword arguments, then you should have matching signatures (and optionally use @overload)

[wesselb]

@gabrieldemarmiesse, I see! If I understand correctly, the idea is to tie argument names to positions, e.g. x should always be the first argument and y the second, and to resolve things that way. I think that should be possible! Perhaps this shouldn't be the default behaviour, as requiring arguments to be named in a specific ways might be undesirable in certain scenarios, but we could allow the user to opt into the behaviour:

from plum import Dispatcher


dispatch = Dispatcher(named_positional=True)


...

Very cool idea! :)

[gabrieldemarmiesse]

Perhaps this "named_positional" should be inferred automatically from the list of signatures? If it's possible then the UX would be better as users wouldn't have to think about it. I'll try to find out if that is possible.

[wesselb]

@gabrieldemarmiesse Right! That sounds even better. I would be fully on board with that. :)

[repst]

I explored integrating plum dispatch into a large existing code base. Since the user would have to know that a function is dispatched in order to call it correctly (e.g. can't assign a positional arg as a keyword arg or vice versa), I was not comfortable making dispatched functions user-facing. Instead, I used the following workaround that hides the dispatched functions from the user and allows dispatching on both positional and keyword arguments:

from plum import dispatch

def user_facing(x, y, z="default"):
    # assign all args and kwargs as positional args in _private
    return _private(x, y, z)

@dispatch
def _private(x: int, y: int, z: tuple):
    pass

@dispatch
def _private(x: str, y: str, z: str):
    pass

Not sure if this is well-known or not.

[wesselb]

Hey @repst! Thanks for sharing this. I think that’s a neat little trick which finds an interesting middle ground between user friendliness and flexibility. :)

[Splines]

Unfortunately with the suggestion by repst, the user won't get nice IntelliSense and see what dispatched versions of a method there are available and what types. Would really appreciate being able to use keyword arguments with the otherwise amazing plum ;)