FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[mitar]

I cannot share one of our datasets (mnist). When running dat share for some time, it dies with:

<--- Last few GCs --->

[26351:0x226d3a0]  1442892 ms: Mark-sweep 1400.9 (1513.8) -> 1400.9 (1513.8) MB, 3027.1 / 28.3 ms  allocation failure GC in old space requested
[26351:0x226d3a0]  1445822 ms: Mark-sweep 1400.9 (1513.8) -> 1400.9 (1463.8) MB, 2924.1 / 18.4 ms  last resort GC in old space requested
[26351:0x226d3a0]  1449445 ms: Mark-sweep 1400.9 (1463.8) -> 1400.9 (1444.8) MB, 3621.0 / 26.1 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xdcf97c25ee1 <JSObject>
    1: _inflate [.../lib/node_modules/dat/node_modules/append-tree/index.js:~619] [pc=0x17fe8be94533](this=0x268a80b5c911 <Tree map = 0x2221e97a87f1>,seq=64875,buf=0x2b408b3b4c61 <Uint8Array map = 0x2cfffc2c1971>)
    3: _list [.../lib/node_modules/dat/node_modules/append-tree/index.js:132] [bytecode=0x1eb2e368da1 offset=13](this=0x268a80b5c911 <Tree map = ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [dat]
 2: 0x121a2cc [dat]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [dat]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [dat]
 5: v8::internal::Factory::NewUninitializedFixedArray(int) [dat]
 6: 0xe1c4cc [dat]
 7: 0xe26bc8 [dat]
 8: 0xe2717b [dat]
 9: v8::internal::JSArray::SetLength(v8::internal::Handle<v8::internal::JSArray>, unsigned int) [dat]
10: v8::internal::ArrayConstructInitializeElements(v8::internal::Handle<v8::internal::JSArray>, v8::internal::Arguments*) [dat]
11: v8::internal::Runtime_NewArray(int, v8::internal::Object**, v8::internal::Isolate*) [dat]
12: 0x17fe8bd0463d
Aborted (core dumped)

I am trying to share a dataset which is 280 MB and has around 70k files.

[mitar]

I am trying now with:

node --max-old-space-size=4096 `which dat` share

[mitar]

Didn't work either. It failed again. Even with more memory given. Not sure why it is requiring so much memory to create metadata. .dat directory is currently 2.1 GB.

[mitar]

I tried adding things to git, it was very quick, memory use was not noticeable, and 292 MB is used by metadata.

[mitar]

Reproduction (similar dataset to one above):

$ git clone https://github.com/myleott/mnist_png.git
$ cd mnist_png
$ rm -rf .git
$ tar -xzf mnist_png.tar.gz
$ cd mnist_png
$ dat create
$ dat share

[joehand]

Thanks for the test dataset, very helpful! Was also slow on my machine.

[mitar]

Something is strange though. So this dataset I linked above has the almost the same content as mine on which dat fails. But now running dat share finished. And .dat directory is only 202 MB. While for my original dataset (where dat crashes with memory issue and does not finish) it is 2.1 GB already.

One difference I see is that in "reproduction" dataset files are split into multiple directories. While in my original dataset all files are in same directory. Does this matter for dat?

[mitar]

I moved all files to one directory using:

find * -type f | xargs -n 1 -I % bash -c 'mv % $(echo % | tr / -)'

It seems this takes now much much much longer. I expect also that it will fail with memory error. So 70k files in a directory is somewhere a threshold.

[mitar]

Full reproduction is:

$ git clone https://github.com/myleott/mnist_png.git
$ cd mnist_png
$ rm -rf .git
$ tar -xzf mnist_png.tar.gz
$ cd mnist_png
$ find * -type f | xargs -n 1 -I % bash -c 'mv % $(echo % | tr / -)'
$ dat create
$ dat share

This might even finish after a long time. But if you then stop and run dat share again, it will try to load all metadata in and die from memory error after some time.

[max-mapper]

I also got this today on a long running dat share:

0 connections | Download 0 B/s Upload 0 B/s

Watching for file updates



Ctrl+C to Exit

<--- Last few GCs --->

1081660890 ms: Mark-sweep 1373.8 (1406.2) -> 1373.7 (1406.2) MB, 4176.8 / 0.0 ms [allocation failure] [GC in old space requested].
1081665196 ms: Mark-sweep 1373.7 (1406.2) -> 1373.7 (1406.2) MB, 4305.9 / 0.0 ms [allocation failure] [GC in old space requested].
1081669530 ms: Mark-sweep 1373.7 (1406.2) -> 1374.7 (1403.2) MB, 4333.4 / 0.0 ms [last resort gc].
1081673058 ms: Mark-sweep 1374.7 (1403.2) -> 1375.7 (1403.2) MB, 3528.6 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3487dd6cf781 <JS Object>
    1: from [native array.js:1334] [pc=0x10d6bc35178e] (this=0x3487dd6b0ae9 <JS Function Array (SharedFunctionInfo 0x3487dd684879)>,bG=0x374c8824bf09 <JS Array[2]>,bH=0x3487dd604381 <undefined>,br=0x3487dd604381 <undefined>)
    2: arguments adaptor frame: 1->3
    3: exec(aka exec) [/home/max/.npm/lib/node_modules/dat/node_modules/wrap-ansi/index.js:114] [pc=0x10d6be68edda] (this=0x3487dd60438...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [dat]
 2: 0x10a03cc [dat]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [dat]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [dat]
 5: v8::internal::Factory::NewTransitionArray(int) [dat]
 6: v8::internal::TransitionArray::Insert(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [dat]
 7: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Handle<v8::internal::LayoutDescriptor>, v8::internal::TransitionFlag, v8::internal::MaybeHandle<v8::internal::Name>, char const*, v8::internal::SimpleTransitionFlag) [dat]
 8: v8::internal::Map::CopyAddDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [dat]
 9: v8::internal::Map::CopyWithConstant(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::TransitionFlag) [dat]
10: v8::internal::Map::TransitionToDataProperty(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [dat]
11: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [dat]
12: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [dat]
13: v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::JSObject::AccessorInfoHandling) [dat]
14: v8::internal::Runtime_DefineDataPropertyInLiteral(int, v8::internal::Object**, v8::internal::Isolate*) [dat]
15: 0x10d6bb8092a7
Aborted (core dumped)