XLSX.jl
XLSX.jl copied to clipboard
Support for large datasets
Hello!
Thank you for this tool! I would like to write large datasets into Excel. Now I have a DataFrame of 5 cols and 600,000 rows and I get an out of memory error. Unfortunately Julia crashed so no further details about the error were available.
XLSX.openxlsx("filename.xlsx", mode="rw") do xf
XLSX.writetable!(xf["sheet1"], collect(DataFrames.eachcol(df)), DataFrames.names(df), anchor_cell=XLSX.CellRef("B5"))
end
Julia 1.5.2 Kubuntu 18.04
Now I recorded the error:
ERROR: XMLError: Memory allocation failed : growing buffer from buffers module (code: 2, line: 0)
Stacktrace:
[1] throw_xml_error() at /home/jube/.julia/packages/EzXML/ZNwhK/src/error.jl:87
[2] macro expansion at /home/jube/.julia/packages/EzXML/ZNwhK/src/error.jl:52 [inlined]
[3] parsexml(::String) at /home/jube/.julia/packages/EzXML/ZNwhK/src/document.jl:80
[4] update_worksheets_xml!(::XLSX.XLSXFile) at /home/jube/.julia/packages/XLSX/vGPvE/src/write.jl:129
[5] writexlsx(::String, ::XLSX.XLSXFile; overwrite::Bool) at /home/jube/.julia/packages/XLSX/vGPvE/src/write.jl:55
[6] openxlsx(::var"#25#26", ::String; mode::String, enable_cache::Bool) at /home/jube/.julia/packages/XLSX/vGPvE/src/read.jl:139
The Julia process consumes some 9 GB memory before crashing. The resulting Excel file would be around 23 MB.