XLSX.jl icon indicating copy to clipboard operation
XLSX.jl copied to clipboard

Support for large datasets

Open Mastomaki opened this issue 2 years ago • 2 comments

Hello!

Thank you for this tool! I would like to write large datasets into Excel. Now I have a DataFrame of 5 cols and 600,000 rows and I get an out of memory error. Unfortunately Julia crashed so no further details about the error were available.

XLSX.openxlsx("filename.xlsx", mode="rw") do xf
        XLSX.writetable!(xf["sheet1"], collect(DataFrames.eachcol(df)), DataFrames.names(df), anchor_cell=XLSX.CellRef("B5")) 
end

Julia 1.5.2 Kubuntu 18.04

Mastomaki avatar Apr 12 '22 09:04 Mastomaki

Now I recorded the error:

ERROR: XMLError: Memory allocation failed : growing buffer from buffers module (code: 2, line: 0)
Stacktrace:
 [1] throw_xml_error() at /home/jube/.julia/packages/EzXML/ZNwhK/src/error.jl:87
 [2] macro expansion at /home/jube/.julia/packages/EzXML/ZNwhK/src/error.jl:52 [inlined]
 [3] parsexml(::String) at /home/jube/.julia/packages/EzXML/ZNwhK/src/document.jl:80
 [4] update_worksheets_xml!(::XLSX.XLSXFile) at /home/jube/.julia/packages/XLSX/vGPvE/src/write.jl:129
 [5] writexlsx(::String, ::XLSX.XLSXFile; overwrite::Bool) at /home/jube/.julia/packages/XLSX/vGPvE/src/write.jl:55
 [6] openxlsx(::var"#25#26", ::String; mode::String, enable_cache::Bool) at /home/jube/.julia/packages/XLSX/vGPvE/src/read.jl:139

Mastomaki avatar Apr 12 '22 15:04 Mastomaki

The Julia process consumes some 9 GB memory before crashing. The resulting Excel file would be around 23 MB.

Mastomaki avatar Apr 13 '22 05:04 Mastomaki