CSV.Chunks splits file into uneven chunks

[aris-mav]

In the following example the "a" variable does not have a consistent size.

using CSV, DataFrames

number_of_lines = 10^6
CSV.write("data.csv", DataFrame(rand(number_of_lines, 10), :auto))

steps = 10

@time for chunk in CSV.Chunks("data.csv"; ntasks=steps)
    a = chunk |> DataFrame
    display(size((a)))
end

Please also have a look at this discourse post, for context.