Support adding to named tuples?

[marius311]

Would it make sense for this package to support adding to named tuples? E.g.

julia> nt = (x = 1, y = 2)
(x = 1, y = 2)

julia> @set! nt.z = 3
(x = 1, y = 2, z = 3)

To me at least it'd be useful functionality that I don't think exists elsewhere.

[jw3126]

You could do:

julia> nt = (x = 1, y = 2)
(x = 1, y = 2)

julia> (;nt..., z= 3)
(x = 1, y = 2, z = 3)

[jw3126]

When we implemented NamedTuple support, we actually thought about this case. I think there was even a short discussion with @tkf about it, but I am not sure. See also this test. I am slightly against, as

• this can be a source of bugs, when you add e.g. a misspelled field instead of updating one etc.
• does not fit well into the lens picture
• (;nt..., z=3) is not much typing.

[tkf]

tl;dr: There is a way to do add/delete for map-like objects using lenses. Whether or not this is a good approach is hard to decide (for me).

• does not fit well into the lens picture

I remember I said a similar thing when we discussed it before. But I just realized that Haskell's lens library actually supports it. The idea is to define a lens that returns a Maybe{T} (:= Union{Some{T}, Nothing}) and use nothing to mean non-existing. We can then use this lens to not only get/set but also to add/delete entries. Translating it to Setfield, it would be something like this

using Setfield
import Setfield: get, set

struct MaybeKeyLens{T} <: Lens
    key::T
end

# get(obj, lens) :: Maybe{T}
get(obj, lens::MaybeKeyLens) =
    haskey(obj, lens.key) ? Some(obj[lens.key]) : nothing

# set(obj, lens, ::Maybe{T})
set(obj, lens::MaybeKeyLens, ::Nothing) = delete(obj, lens.key)
set(obj, lens::MaybeKeyLens, val::Some) = setkey(obj, lens.key, something(val))

where delete and setkey has to be defined elsewhere.

For Dict they are:

function delete(obj::Dict, key)
    clone = copy(obj)
    pop!(clone, key, nothing)
    return clone
end

function setkey(obj::Dict, key, val)
    clone = copy(obj)
    clone[key] = val
    return clone
end

For NamedTuple,

using Setfield: PropertyLens

delete(obj::NamedTuple{names}, key) where names =
    NamedTuple{Tuple(n for n in names if n !== key)}(obj)

setkey(obj::NamedTuple{names}, key, val) where names =
    if key in names
        set(obj, PropertyLens{key}(), val)
    else
        NamedTuple{(names..., key)}(obj..., val)
    end

Type stable version

You obviously needs a generated function :)

@generated _remove(::Val{xs}, ::Val{x}) where {xs, x} =
    :($(Tuple(y for y in xs if y != x)))

delete(obj::NamedTuple{names}, key) where names =
    NamedTuple{_remove(Val(names), Val(key))}(obj)

Examples:

julia> get((a=1, b=2), MaybeKeyLens(:a))
Some(1)

julia> get((a=1, b=2), MaybeKeyLens(:c))

julia> set((a=1, b=2), MaybeKeyLens(:a), Some(100))
(a = 100, b = 2)

julia> set((a=1, b=2), MaybeKeyLens(:a), nothing)
(b = 2,)

If we go to this route, I think this point

• this can be a source of bugs, when you add e.g. a misspelled field instead of updating one etc.

is more or less irrelevant as we wouldn't be touching the normal IndexLens and PropertyLens. (You need to be as careful as when manipulating Dict usually.)

But the third point

• (;nt..., z=3) is not much typing.

is still true. On one hand, MaybeKeyLens let us abstract out operations on any map-like objects like NameTuple and AbstractDict so it sounds like a good feature to have. On the other hand, I'm not quite sure if this a right tool in Julia because I don't think you need to frequently write a function that has to work with AbstractDict and NamedTuple. There are other problems:

• It's hard to come up with a good sugar (with a valid syntax). One possibility is @lens @? _[:a] which can also be written as @lens@? _[:a] so that @? looks like a suffix. (Off topic, but it may actually be useful since we can define all the mutating variants of all lenses with @lens@! _.a etc. But this syntax is not super pretty...)

• To use this with AbstractDict, we need either an in-place variant of this lens or a good persistent dictionary.

• Post-composing normal lenses do not work. That is to say, get((a=(b=1,),), MaybeKeyLens(:a) ∘ @lens _.b) would be an error. I think you'd need some collections of higher-order functions to automatically "lift" the lenses etc.

[jw3126]

Thanks for the thorough analysis! I think from a purely mathematical perspective your suggestion is very natural. Using Maybe one can add fields to a NamedTuple without violating the lens laws. I also like your implementation snippets and agree with everything you said. Overall I am not convinced. I think syntax + composability of this adds too much complexity for too little gain. But this might be personal bias. I don't add to NamedTuples very often. If somebody knows a practical example where such a lens shines, please share it.

Also I think there are some differences between julia and Haskell that make this lens more attractive in the latter:

• In Haskell there are no julia style exceptions. One has to use Maybe (or fancy variants of it) for operations that would throw e.g. a KeyError in julia. So this lens feels more idomatic in Haskell and fits nicely with the Map API there.
• Haskell has static type checking, that makes it easier to do fancy compositions and lifts correctly.

[tkf]

I think syntax + composability of this adds too much complexity for too little gain.

Yeah, it makes sense. I really like that Setfield is very minimalistic but yet super powerful and also extensible. Adding this feature could ruin it. I agree that "wait for a practical example" for this lens is the right approach. (I'm supposing that the original request for "adding a field to a NamedTuple" is not a big enough motivation as there is a native syntax and merge in Julia.)

• Haskell has static type checking, that makes it easier to do fancy compositions and lifts correctly.

It's a bit tangential and maybe nit-picky, but I think support for the higher-kinded types has a bigger impact here. Run-time type assertion is not so crazy and maybe we would have static type checker in the future (at least JavaScript and Python pulled it off). But inability to express something like Functor feels limiting (That was my impression after writing Haskell-style functor-based lens in Setfield framework) and fundamental (as tools or practices can't fix it). Or maybe it's not a big problem? I guess I need an implementation of such lifting functions in Julia to really see it. Or maybe there is a way around the lack of native higher-kinded types, like the trait system is implemented in the "user land?"

[jw3126]

I agree that "wait for a practical example" for this lens is the right approach. (I'm supposing that the original request for "adding a field to a NamedTuple" is not a big enough motivation as there is a native syntax and merge in Julia.)

Yes. In the initial version of Setfield I was not convinced that allowing a lens to change the type of an object was a good idea. Then you came up with using lenses to specify a partial derivative and that completely changed my mind. If there is such a killer application here as well, that could justify the introduction of new syntax.

It's a bit tangential and maybe nit-picky, but I think support for the higher-kinded types has a bigger impact here.

Probably yes. I think the problems start at a less sophisticated level already. In Julia a function does not have a precise domain and range. E.g. we only have Function not Function{S,T} (This might be the price of multiple dispatch, not sure).

[tkf]

Ah yes, Function{S,T} is pretty essential too.