[BUG] Exporting a big file (>50MB) produces a V8 memory heap error

[MKCG]

Describe the bug When a report is generated containing a lot of document then the Javascript v8 engine might suffers from a heap out of memory exception and crash.

I believe that it is caused by the fact that the export process keeps all content that should be exported in memory before writing all at once into the CSV file.

To Reproduce Steps to reproduce the behavior:

1. Go to 'Discover'
2. Select a lot of documents and do not specify the columns that should be exported
3. Click on 'Generate CSV'
4. The dashboard application should crash

Expected behavior Their should be almost no limit to the size of the exported CSV file and it should be possible to generate CSV file way bigger than the memory allocated to the Javascript heap.

Example of a trace

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[1:0x41f1b90]  1220660 ms: Mark-sweep 786.3 (1233.8) -> 786.3 (1157.3) MB, 218.8 / 0.0 ms  (average mu = 0.853, current mu = 0.000) last resort GC in old space requested
[1:0x41f1b90]  1221123 ms: Mark-sweep 786.3 (1157.3) -> 786.3 (1116.8) MB, 462.8 / 0.0 ms  (average mu = 0.630, current mu = 0.000) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x16ab3e6dbf1d]
Security context: 0x3a9704a9e6c1 <JSObject>
    1: DoJoin(aka DoJoin) [0x3a9704a85e69] [native array.js:~87] [pc=0x16ab3eace969](this=0x07a6e6d826f1 <undefined>,l=0x0a2acd302271 <JSArray[10000]>,m=10000,A=0x07a6e6d828c9 <true>,w=0x0a66bb934901 <String[1]\: \n>,v=0x07a6e6d829a1 <false>)
    2: Join(aka Join) [0x3a9704a85eb9] [native array.js:~112] [pc=0x16ab3eaaa4b8](this=0x07a6e6d826f1 <undefined>,l=0...

 1: 0x8fb090 node::Abort() [/usr/share/kibana/bin/../node/bin/node]
 2: 0x8fb0dc  [/usr/share/kibana/bin/../node/bin/node]
 3: 0xb0336e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
 4: 0xb035a4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
 5: 0xef7602  [/usr/share/kibana/bin/../node/bin/node]
 6: 0xf06e1f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/share/kibana/bin/../node/bin/node]
 7: 0xed69cb v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
 8: 0x11c5768 v8::internal::Runtime_StringBuilderJoin(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/bin/../node/bin/node]
 9: 0x16ab3e6dbf1d 

Host/Environment (please complete the following information):

• OS: Ubuntu
• Version: 20.10
• Docker version : 19.03.23
• Docker image : amazon/opendistro-for-elasticsearch-kibana:1.13.2

[zhongnansu]

Thx for reporting this bug. We did need a limit on the size of exportable csv to avoid out of memory issue. Maybe configure this as a plugin config item, with a default value.

[MKCG]

Technically I don't think there is a necessity to define a size limit.

The current issue is due to the fact that the CSV is created only after having retrieved all the matching data. However, with a streaming based implementation that should not be an issue at all.

I made a quick implementation on the following branch but I have not tested it nor finalized it yet : https://github.com/MKCG/dashboards-reports/pull/1/files

May I open a draft pull request on this repository to suggest a streaming implementation ?

[zhongnansu]

Technically I don't think there is a necessity to define a size limit.

The current issue is due to the fact that the CSV is created only after having retrieved all the matching data. However, with a streaming based implementation that should not be an issue at all.

I made a quick implementation on the following branch but I have not tested it nor finalized it yet : https://github.com/MKCG/dashboards-reports/pull/1/files

May I open a draft pull request on this repository to suggest a streaming implementation ?

Yes, a draft PR will be better. Thx

[MKCG]

Ok, I will do that maybe next week.