BIDS validator error while uploading through CLI

[Arshitha]

What went wrong?

While trying to upload a BIDS dataset using the openneuro cli I'm getting following error at the BIDS validation stage:

[arshithab@helix BIDS_defaced]$ openneuro upload --ignoreWarnings .
? This will create a new dataset, continue? Yes
? Please affirm one of the following: All structural scans have been defaced, obscuring any tissue on or near the face that could potentially be used to reconstruct the facial structure.
[email protected]

<--- Last few GCs --->

[2560486:0x69be410]   143041 ms: Scavenge 4066.6 (4098.7) -> 4059.1 (4098.7) MB, 1.04 / 0.00 ms  (average mu = 0.213, current mu = 0.220) allocation failure; 
[2560486:0x69be410]   143055 ms: Scavenge 4066.6 (4098.7) -> 4059.1 (4098.7) MB, 1.03 / 0.00 ms  (average mu = 0.213, current mu = 0.220) allocation failure; 
[2560486:0x69be410]   143070 ms: Scavenge 4066.6 (4098.7) -> 4059.1 (4098.7) MB, 1.00 / 0.00 ms  (average mu = 0.213, current mu = 0.220) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc9e850 node::Abort() [node]
 2: 0xb720ff  [node]
 3: 0xec1a70 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0xec1d57 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 5: 0x10d3dc5  [node]
 6: 0x10d4354 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0x10eb244 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
 8: 0x10eba5c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0x10c1d61 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0x10c2ef5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0x109f536 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
12: 0x1090b6c v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [node]
13: 0x1090cd4 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [node]
14: 0x10ad924 v8::internal::Factory::NewJSArrayStorage(v8::internal::ElementsKind, int, v8::internal::ArrayStorageAllocationMode) [node]
15: 0x10ada3e v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllocationMode, v8::internal::AllocationType) [node]
16: 0x15121df v8::internal::Runtime_StringSplit(int, unsigned long*, v8::internal::Isolate*) [node]
17: 0x1934ef6  [node]
Aborted (core dumped)

Expected behavior

Upload should go through without error

How to reproduce

No response

Desktop

• OS: Linux
• Browser:
• OpenNeuro CLI version: 4.21.2

Phone

• Device:
• OS:
• Browser:
• Version:

Additional information

I tried validating the dataset using the latest BIDS validator (separately) and got the same error. I'm yet to try uploading through the browser though.

[Arshitha]

Hi @nellh or @effigies - Just wanted to check-in on this if you'd had a chance to look into it. I wasn't able to upload from the browser either and I've tried using older versions of openneuro cli but I still get the same error as above.

Thank you very much in advance! Arsh

[Arshitha]

Looks like the issue was because of the beh/ files in the BIDS tree. Adding them to .bidsignore worked but that doesn't explain why it led to the above error...

[effigies]

It appears to be an out-of-memory issue. Did you have a lot of behavioral files?

You can override the default heap size with:

NODE_OPTIONS="--max-old-space-size=8192" openneuro ...

[Arshitha]

There were about 610 behavioral files. Would that be considered large by the bids-validator? Thanks! I'll test it with the override option and see how it goes!