Memory problems and long load times for multiframe ultrasound data

[dereklukacs]

Hello @dannyrb you seem most active on this problem, after several days of debugging I too am having issues loading Multiframe Ultrasound data. I have compiled a list of related issues from this repo at the bottom.

In my case I have RGB encoded, 400mb, 204 frame Ultrasound data in a single multiframe DCM file with transfer syntax 1.2.840.10008.1.2.1

Config:

var config = {
  maxWebWorkers: 1,
  startWebWorkersOnDemand: false,
  taskConfiguration: {
    decodeTask: {
      initializeCodecsOnStartup: false,
      loadCodecsOnStartup: false,
      usePDFJS: false,
      strict: false,
    },
  },
  webWorkerTaskPaths: [
    'https://unpkg.com/[email protected]/dist/610.bundle.min.worker.js',
    'https://unpkg.com/[email protected]/dist/888.bundle.min.worker.js',
  ],
};

A few observations:

• The per-frame load time (after the network call finishes) appears to scale with the size of the source file (dataset). I experience near-instant loads with a 12 frame file. Intermediate per-frame load times for a 100mb fluoroscopy file. ~1 sec / frame load times for the 400 mb file described above.
• If I throttle the decoding to the above configuration (load/initializeCodecs: false, maxWebWorkers:1) and only request one frame at a time before requesting for another (no queueing in webWorkerManager) then usually I can load all frames (this takes several minutes). Sometimes though the garbage collection seems to stop working and the memory will quickly climb to take up everything on my machine (16gb available, crashes around 15gb).
• After crashing, the memory will never get cleared
• Manually hitting the garbage collection button on chrome dev tools wipes this 15gb down to <1gb.
• running a profile on loading indicates that ~97% of the time is spent doing postMessage, seems like a lot of overheard for using webworkers.

Conclusions/Deductions from above observations

• For each frame request, a fully copy of the source dataset is being made
• The data is freed, but for some reason not being properly garbage collected, even after a very long time

What does a solution look like:

1. A fix to reduce memory allocation and time taken to load/decode a single frame
2. A bulk decode method that decodes all of the frames, even it is blocking.
3. A way to decode without the use of webworkers (and without allocating new memory)

Related issues

#235 #425 #411 https://github.com/cornerstonejs/cornerstoneWADOImageLoader/issues/156#issuecomment-794628050 https://github.com/cornerstonejs/cornerstoneWADOImageLoader/issues/428 #373

https://github.com/cornerstonejs/cornerstone/issues/576 https://github.com/cornerstonejs/cornerstone/issues/519

[nullhook]

i'd do this progressively using wasm/emscripten. Your last two points can be solved using pthreads to decode all frames which would be done off the main browser thread (non-blocking) and in-effect it would use less memory.

[dereklukacs]

Thanks for the reply @nullhook, I am not sure I understand what you mean though. Isn't this package already running everything on wasm?

[nullhook]

i believe it uses web workers to decode frames not wasm.

based on this file it looks like wasm port is in progress and doesn't work yet.

edit: I do see some decoders with wasm imports. however, i still think this can be improved if frames were decoded on the gpu

[Ouwen]

@dereklukacs do you have the file in question to upload?

[nyacoub]

This is related to issue #71 To free the worker memory, I use this code cornerstoneWADOImageLoader.webWorkerManager.terminate();

[dereklukacs]

@Ouwen I cannot share unfortunately.

@nyacoub Yes that seems precisely related and we have also noticed it being worse in chrome. I'll try your proposed workaround. Thanks!

[Ouwen]

#454

@dereklukacs does this PR fix your issue?

[dereklukacs]

Hi @Ouwen I'll try this out today, based on the description it sounds like it should fix the issue.

[dereklukacs]

@nyacoub

This seems to help fix the runaway memory problems, thanks!

[muhammedmokbel]

@dereklukacs @Ouwen
could you please let me know how can I use this solution to solve out of memory problem ?

[Ouwen]

Hi @muhammedmokbel you'll need to run the following steps.

1. Clone this repo with the above fix
2. Run npm run build within the root of this repo
3. Run yarn link within the root of this repo
4. Run yarn link cornerstone-wado-image-loader in the Viewer or the repo(s) you are working in.

[muhammedmokbel]

sorry for being late @Ouwen thank you so much

[chengfeng311]

This is related to issue #71 To free the worker memory, I use this code cornerstoneWADOImageLoader.webWorkerManager.terminate();

@nyacoub could you please tell us where we can use this code?

[nyacoub]

@chengfeng311 use this after retrieving the image. It will kill the workers and free up the memory. In some cases, the memory usage will is high due to the encoder's workers, but the code above will free up that memory and the browser will not crash!

[chengfeng311]

Thanks for reply, but which not solve my problem, I terminate webworker after loaded all images, but when i switch image with cornerstone.loadImage, memory still raise 2%. Do you have any way to solve this problem? @nyacoub

[Ouwen]

Might need to use chrome memory profiler to take a look