evalv3: 8X memory usage increase

[alecholmez]

What version of CUE are you using (cue version)?

$ cue version v0.10.0

Does this issue reproduce with the latest stable release?

Yes this is reproducible with 0.10.0.

What did you do?

We use the go API internally to evaluate CUE projects. With evalv2 enabled, evaluating 3 projects at once, we see about a 2.3gb mem usage ceiling. With the exact same workload of 3 projects, we saw a spike of about 18gb of mem.

Similarly, evalv2 yields a CPU usage of about 1.2 cores max. evalv3 uses about 2.5 cores.

This code is quite custom so would be more than happy to setup a meeting and walkthrough what's going on here.

What did you expect to see?

The same, ideally less memory usage.

What did you see instead?

A massive increase of resource utilization.

[mvdan]

Out of curiosity, how does wall time compare between v2 and v3?

[alecholmez]

@mvdan just me quickly counting, it was negligible. They were almost identical. It did take v3 much longer to garbage collect though. Evaluation time was extremely similar

[alecholmez]

Dupe filed here: https://github.com/cue-lang/cue/issues/3493

Tried again with master:

	cuelang.org/go v0.11.0-alpha.3.0.20241010142549-44cceca80690

FWIW I noticed v3 evaluator was much faster this time around, but still used a seriously large amount of memory. GC behavior was similar but I can forcefully trigger a run of the garbage collector so that's not a major issue for us.

[mvdan]

@alecholmez would you be able to try again with the latest master, and post wall time and memory usage numbers?

Also, is this with a public project or with a project that you shared with us via Unity? That would let us investigate further.

[mvdan]

This code is quite custom so would be more than happy to setup a meeting and walkthrough what's going on here.

I would also be up for this if that's the best route for you - perhaps in the new year? DM me if so.

[alecholmez]

@mvdan When trying to evaluate with V3 I am now seeing this error which I did not see before:

error in call to list.Concat: not properly initialized (state: unprocessed, value: *adt.Vertex)

A brief snippet of the code that triggers this:

	things: [...api.#Thing]

	_default: [#api.Thing]
	if do_default {
		_default: [
			{
                                .. the schema for api.#Thing with concrete values
			},
		]
		redirects: list.Concat([_default, things])
	}

I would also be up for this if that's the best route for you - perhaps in the new year? DM me if so.

This is a new project that has not currently been registered in unity but the CUE is what I have sent over to you in the past. I'd be more than happy to hop on a call if you'd like to set that up.

[mvdan]

Thank you - I've actually run into that panic myself just today as well, so I'll look at reducing and filing that bug.

[mvdan]

Forgot to update this thread - that bug filed is now https://github.com/cue-lang/cue/issues/3648.

[mvdan]

This issue affected the same project which now fails due to an error reduced in https://github.com/cue-lang/cue/issues/3850. So we will come back to this once 3850 is resolved, to see what memory usage is like afterwards.

[mxey]

So we will come back to this once 3850 is resolved, to see what memory usage is like afterwards.

3850 is resolved now.

[mvdan]

Indeed. We are looking into memory usage shortly, along with https://github.com/cue-lang/cue/issues/3334. We will post updates once we have any.

[mvdan]

Hi again @alecholmez @mxey - could you please try master again and report how memory usage looks for you now? I just posted an update at https://github.com/cue-lang/cue/issues/3334#issuecomment-3057186530.

[mxey]

could you please try master again and report how memory usage looks for you now?

Very happy to report it looks much better now. Down from 3.8 GB to 0.8 GB. evalv2 was at 0.6 GB.

$ CUE_EXPERIMENT=evalv3=0 /usr/bin/time -l cue vet -c 2>&1 | grep "peak memory footprint"
           569788672  peak memory footprint
$ /usr/bin/time -l cue vet -c 2>&1 | grep "peak memory footprint"               
          3803832576  peak memory footprint
$ /usr/bin/time -l ~/go/bin/cue vet -c 2>&1 | grep "peak memory footprint"
           805669568  peak memory footprint

[jack-bischoff]

Based on some cue export tests, we see about a 3x improvement in memory usage from 0.13. Which is tremendous progress so good work all around! We will continue to test the Go API within our full stack but I reckon we will see similar results.

[mvdan]

Great to hear! Let us know if memory usage is acceptable as of master. We're not looking for perfect right now - there is still work to be done - but we'll be happy enough if memory usage is not significantly worse than the old evaluator.

[mvdan]

@alecholmez also confirmed that the memory usage they see with master is much improved compared to evalv2, so I'll consider this issue resolved now :)

More incremental improvements with memory usage are coming, but we believe the bulk of it has already landed.