MWE for the crash

[drozzy]

Ok, finally figured out what cases #14 to crash:

using DataLoaders

features = rand(2)
dataloader = DataLoader((features, features), 100, useprimary = true)

for (xs, ys) in dataloader
    print(size(xs))
    print(size(ys))
    break
end

You will see it crashes during the loop, and then hangs inside of it.

[drozzy]

Base.StackTraces.StackFrame[(::DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}})(::Int64) at workerpool.jl:55, macro expansion at workerpool.jl:65 [inlined], #6 at macros.jl:19 [inlined], #63 at qpool.jl:195 [inlined], (::ThreadPools.var"#59#60"{ThreadPools.var"#63#65"{DataLoaders.var"#6#12"{DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}}}},Tuple{Int64,Int64}})() at qpool.jl:86]
┌ Error: Exception while executing task on worker 4. Shutting down WorkerPool.
│   e =
│    MethodError: no method matching copyrec!(::SubArray{Float64,0,Array{Float64,1},Tuple{Int64},true}, ::Float64)
│    Closest candidates are:
│      copyrec!(::AbstractArray, ::AbstractArray) at /home/andriy/.julia/packages/DataLoaders/uGlPg/src/batchview.jl:112
│   stacktrace = 6-element Array{Base.StackTraces.StackFrame,1}: …
│   args = 1
└ @ DataLoaders ~/.julia/packages/DataLoaders/uGlPg/src/workerpool.jl:56

[drozzy]

I mean I solved it by reshaping the input to be (1, n).

But I think the error could be more helpful and the hanging makes it necessary to kill Julia process.

[darsnack]

Okay seems like this bug is an MLDataPattern.jl issue. Iterating a row vector gives arrays, but iterating a column vector gives scalars. We should specialize getobs on AbstractVectors separately from AbstractArrays (right now they hit the same method).

[lorenzoh]

@darsnack Should we open an issue in MLDataPattern.jl then?

@drozzy Unfortunately I haven't been able to reliably send worker exceptions to the main thread so it is interrupted. This line is supposed to do just do that, but doesn't do so reliably.