Allow reference to previously defined columns in @transform

[fredcallaway]

This might be too big of an ask, but in dplyr we can do this

data.frame(x=c(1,2,3)) %>% mutate(y=2*x, z=y+2)

that is, defining z and then using it in the same mutate call.

But is not possible with @transform. It would be very nice to be able to do

@transform DataFrame(x=[1,2,3]) begin
    :y = :x .* 2
    :z = :y .+ 2
end

Is there a reason this isn't supported?

[bkamins]

The reason is that, in general, operations:

:y = :x .* 2

and

:z = :y .+ 2

could be executed in parallel, so :y is not yet present when you want to compute :z.

However, DataFramesMeta.jl could split this single @transform call into multiple transform calls to get what you want (but it requires a decision if we want it).

[pdeffebach]

Yes, this would be nice. However DataFramesMeta.jl is restricted by how DataFrames.transform works.

There is a work-around using the @astable macro flags. Think of @astable as a nice way to make multiple inter-dependent columns, as well as define multiple intermediate variables. In the latter sense, it's over kill for what you want, which is just to make columns. S you can do this:

julia> @transform DataFrame(x=[1,2,3]) @astable begin
           :y = :x .* 2
           :z = :y .+ 2
       end
3×3 DataFrame
 Row │ x      y      z     
     │ Int64  Int64  Int64 
─────┼─────────────────────
   1 │     1      2      4
   2 │     2      4      6
   3 │     3      6      8

as well as

julia> @transform DataFrame(x=[1,2,3]) @astable begin
           :y = :x .* 2
           a = 5
           :z = (:y .+ 2) .* a
       end
3×3 DataFrame
 Row │ x      y      z     
     │ Int64  Int64  Int64 
─────┼─────────────────────
   1 │     1      2     20
   2 │     2      4     30
   3 │     3      6     40

We don't have @astable as default because it allows for too many things, like intermediate variables, but also makes it so you have to keep track of how you type column names more carefully. The following will error

julia> @transform DataFrame(x=[1,2,3]) @astable begin
           :y = :x .* 2
           :z = $"y" .+ 2
       end

So when you need interdependent variables, I recommend wither @astable or just two @transform calls.

Final note: You are aware of @rtransform, right? Which makes row-wise operations easier.

[fredcallaway]

Ah that's great; I didn't realize @astable had this effect. Perhaps worth adding/modifying an example in the docs to illustrate this; the current example only shows the temporary variable, not using previously defined columns?

I do wonder about @bkamins' suggestion of implementing @transform with multiple transform calls. But perhaps this a substantial performance cost---as far as I know DataFrames doesn't try to parallelize within transform calls, so that doesn't seem like an issue to me.

And yes, I have m-tab bound to @rtransform (m is a habit from dplyr).

[pdeffebach]

yeah a doc fix might be good. Do you want to start a PR (It's not a big deal, though, I can do it!)

I don't think I'm too concerned about the performance cost. Interested parties can always write a transform call. I'm not concerned about code complexity and maintenance burden.

Not to put the burden on @bkamins , but a keyword argument in transform and select might be what I want. But if he doesn't want to add it, I'm happy with the status quo.

I do think @astable is an underrated "killer feature" relative to dplyr, so I should probably evangelize it more.

[bkamins]

But if he doesn't want to add it, I'm happy with the status quo.

It is not the issue of the burden, as we already have threads keyword argument that controls that. The issue is just that I was afraid that changing the parsing rules depending on the value of this keyword argument would be a bit risky.

[fredcallaway]

Ah, I wasn't aware that transform was threaded by default. That's neat! In that case I retract the request; it definitely doesn't make sense to have different semantics based on the threads keyword.

I do think adding an example of using a previously defined column using @astable would be great though.

[fredcallaway]

I don't think this really warrants setting up a fork. Here's a suggestion to add to the docs:

You can also refer to previously defined columns within an `@astable` block.

julia> @transform df @astable begin 
    :c = :b .+ :a
    :d = :c .^ 2
    :d .-= maximum(:d)
end

Note that this is not possible in a bare `@transform` statement because
the separate columns might be created in parallel by separate threads (see [DataFrames Multithreading-support](https://dataframes.juliadata.org/stable/lib/functions/#Multithreading-support)).

You can use arbitrary julia syntax within @astable including other macros and control flow.
Note that we are using `@rtransform` here to treat each row independently.

julia > @rtransform df @astable begin 
    slow(x) = sleep(0.1 * x)
    :runtime = @elapsed slow(:a)

    bad_function(x) = iseven(x) ? error("odd!") : "even"
    :c = try
        bad_function(:a)
    catch
        missing
    end
end

However, one important gotcha is that all new columns must be defined at the top level.
So, the following code would not work

@rtransform df @astable begin 
    if iseven(:a)
        :parity = "even"
    else
        :parity = "odd"
    end
end

This can be fixed by moving the `:parity =` before the `if` (like we did with the try block example)
or by initializing the variable, e.g. `:parity = missing` before the if statement.