openneuro
openneuro copied to clipboard
BIDS validator error while uploading through CLI
What went wrong?
While trying to upload a BIDS dataset using the openneuro cli I'm getting following error at the BIDS validation stage:
[arshithab@helix BIDS_defaced]$ openneuro upload --ignoreWarnings .
? This will create a new dataset, continue? Yes
? Please affirm one of the following: All structural scans have been defaced, obscuring any tissue on or near the face that could potentially be used to reconstruct the facial structure.
[email protected]
<--- Last few GCs --->
[2560486:0x69be410] 143041 ms: Scavenge 4066.6 (4098.7) -> 4059.1 (4098.7) MB, 1.04 / 0.00 ms (average mu = 0.213, current mu = 0.220) allocation failure;
[2560486:0x69be410] 143055 ms: Scavenge 4066.6 (4098.7) -> 4059.1 (4098.7) MB, 1.03 / 0.00 ms (average mu = 0.213, current mu = 0.220) allocation failure;
[2560486:0x69be410] 143070 ms: Scavenge 4066.6 (4098.7) -> 4059.1 (4098.7) MB, 1.00 / 0.00 ms (average mu = 0.213, current mu = 0.220) allocation failure;
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xc9e850 node::Abort() [node]
2: 0xb720ff [node]
3: 0xec1a70 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0xec1d57 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
5: 0x10d3dc5 [node]
6: 0x10d4354 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
7: 0x10eb244 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
8: 0x10eba5c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
9: 0x10c1d61 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0x10c2ef5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0x109f536 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
12: 0x1090b6c v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [node]
13: 0x1090cd4 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [node]
14: 0x10ad924 v8::internal::Factory::NewJSArrayStorage(v8::internal::ElementsKind, int, v8::internal::ArrayStorageAllocationMode) [node]
15: 0x10ada3e v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllocationMode, v8::internal::AllocationType) [node]
16: 0x15121df v8::internal::Runtime_StringSplit(int, unsigned long*, v8::internal::Isolate*) [node]
17: 0x1934ef6 [node]
Aborted (core dumped)
Expected behavior
Upload should go through without error
How to reproduce
No response
Desktop
- OS: Linux
- Browser:
- OpenNeuro CLI version: 4.21.2
Phone
- Device:
- OS:
- Browser:
- Version:
Additional information
I tried validating the dataset using the latest BIDS validator (separately) and got the same error. I'm yet to try uploading through the browser though.
Hi @nellh or @effigies - Just wanted to check-in on this if you'd had a chance to look into it. I wasn't able to upload from the browser either and I've tried using older versions of openneuro cli but I still get the same error as above.
Thank you very much in advance! Arsh
Looks like the issue was because of the beh/
files in the BIDS tree. Adding them to .bidsignore worked but that doesn't explain why it led to the above error...
It appears to be an out-of-memory issue. Did you have a lot of behavioral files?
You can override the default heap size with:
NODE_OPTIONS="--max-old-space-size=8192" openneuro ...
There were about 610 behavioral files. Would that be considered large by the bids-validator? Thanks! I'll test it with the override option and see how it goes!