sparklyr
sparklyr copied to clipboard
sparklyr doesn't work with sparkxgb
After loading sparkxgb (0.1.0), sparklyr (v1.0.1) can't connect to spark (via livy). It failed in the following piece of code in function spark_connect:
if (length(scon$extensions) > 0) { for (initializer in scon$extensions$initializers) { if (is.function(initializer)) initializer(scon) } }
where Browse[4]> scon$extensions [1] "sparkxgb" Browse[4]> class(scon$extensions) [1] "character"
So scon$extensions$initializers throws exception of "$ operator is invalid for atomic vectors".
The same issue occurs when loading the "sparklyr.nested" package prior to connecting to spark.