`UndefVarError` when sending a REPL-defined function to workers with `@eval @everywhere`

[draftman9]

Julia Version: 1.10.5+0.x64.linux.gnu (installed by juliaup) OS/Architecture: Linux x86_64, Ubuntu 22.04

Summary:

Attempting to broadcast a function object defined in the main process's REPL to worker processes using @eval @everywhere f_every = $f results in an UndefVarError: #f not defined on the workers. This suggests an issue with serializing and deserializing a REPL-defined function.

Minimal Working Example (MWE):

using Distributed
addprocs(2)

# Define a function on the main process
f(a) = a

# Attempt to send the function object to all workers
@eval @everywhere f_every = $f

Expected Behavior:

The function f should be successfully serialized, sent to the worker processes, and assigned to the variable f_every on each worker without error.

Actual Behavior:

The command fails with an error on the worker processes:

ERROR: On worker 2:
UndefVarError: `#f` not defined
Stacktrace:
  [1] deserialize_datatype
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:1399
  [2] handle_deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:867
  [3] deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814
  [4] handle_deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:874
  [5] deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814
  [6] deserialize_expr
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:1291
  [7] handle_deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:894
  [8] deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814
  [9] deserialize_expr
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:1291
 [10] handle_deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:894
 [11] deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814
 [12] #5
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:973
 [13] ntupleany
    @ ./ntuple.jl:43
 [14] deserialize_tuple
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:973
 [15] handle_deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:857
 [16] deserialize
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814 [inlined]
 [17] deserialize_msg
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Distributed/src/messages.jl:87
 [18] #invokelatest#2
    @ ./essentials.jl:892 [inlined]
 [19] invokelatest
    @ ./essentials.jl:889 [inlined]
 [20] message_handler_loop
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:176
 [21] process_tcp_streams
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:133
 [22] #103
    @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:121

...and 1 more exception.

Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:455
 [2] macro expansion
   @ ./task.jl:487 [inlined]
 [3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr)
   @ Distributed /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Distributed/src/macros.jl:219
 [4] top-level scope
   @ /media/Big/replaceC/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Distributed/src/macros.jl:203
 [5] eval(m::Module, e::Any)
   @ Core ./boot.jl:385
 [6] top-level scope
   @ REPL[5]:1

[Seelengrab]

I think the usual pattern for this is simply

@everywhere f(a) = a

The reason you get this error is because @everywhere executes the expression you give it on the remote process. As there is no function f defined there, it cannot be assigned to f_every. In general, variables in julia should be thought of as labels or post-it notes attached to objects, not the objects themselves. The macro itself doesn't know which f you'd want to refer to if there already is an f defined on the remote process.

[draftman9]

I knew this usage. Actually I am writing a julia pkg that would be used to simplify the progress to parallel same-kind tasks. The main function of the pkg need to get the custom func and broadcast the func to all workers/processors. Like codes below:

using Distributed
addprocs(2)

# Define a custom function on the main process
f(a) = a
# main function in the pkg
function pkg_main_f(func:Function, arg)
    # Attempt to send the function object to all workers
    @eval @everywhere f_every = $func
    # ... codes about running f_every
end