Performance and scaling issue

[eshubin]

Hello Stefan, I created a timoni manifest for a project with pretty big number of services (in terms of Service Oriented Architecture) and reached some limitations of timoni. As I can not disclose the real source code, a simple demo project is opensourced. In order to reduce a boilerplate code I extracted common pieces of Deployments, ConfigMaps and service configuration. Services (1,2,3 ...) configuration have some repeatable parts such as MongoDB, S3, MySQL, etc. Each of these parts has global and per service configuration, merged in common code. I noticed that with a growth of the manifest a simple command timoni mod vet started taking more and more time and memory. Metrics of memory usage and time were taken from the output of the command /usr/bin/time -v timoni mod vet on Ubuntu 22.04 and timoni 0.24.0 (the same behaviour on older versions), fields Maximum resident set size and Elapsed (wall clock) time correspondingly. As you can see from README on each git tag memory usage grows exponentially for each new service. So for 4 services timoni mod vet used 275MB of memory and took 3 seconds. For 5 it was 400MB and 8 seconds. For 6 it is already 580MB and 10 seconds. On real project with more services and bigger configs I reached 7GB of memory usage and 7 minutes of execution time.

Could you, please, assist with resolving that issue? Best, Eugene

[stefanprodan]

@eshubin can you please build the module using the cue binary as showed here, this will help determine if the issue is in Timoni or in CUE itself.

[eshubin]

cue cmd -t debug -t name=test -t namespace=test -t mv=1.0.0 -t kv=1.28.0 build produces many errors like

timoni.instance.objects.cm6.#config.image: field not allowed: ./templates/service6.cue:9:28 ./debug_values.cue:8:2 ./templates/common.cue:121:17 ./templates/config.cue:9:10 ./templates/config.cue:94:10 ./templates/config.cue:112:14 ./templates/config.cue:112:45 ./templates/service6.cue:3:21 ./templates/service6.cue:4:14 ./timoni.cue:14:9 ./timoni.cue:24:12 ./timoni.cue:27:11 ./timoni.cue:29:11 timoni.instance.objects.cm6.#config.resources: field not allowed: ./templates/service6.cue:9:28 ./debug_values.cue:21:2 ./templates/common.cue:121:17 ./templates/config.cue:9:10 ./templates/config.cue:94:10 ./templates/config.cue:112:14 ./templates/config.cue:112:45 ./templates/service6.cue:3:21 ./templates/service6.cue:4:14 ./timoni.cue:14:9 ./timoni.cue:24:12 ./timoni.cue:27:11 ./timoni.cue:29:11

cue version

cue version v0.12.0

go version go1.23.5 -buildmode exe -compiler gc -trimpath true CGO_ENABLED 0 GOARCH amd64 GOOS linux GOAMD64 v1 cue.lang.version v0.12.0

Cue version is the same as is used in timoni.

[stefanprodan]

You need to set all the required values in the debug values file

[eshubin]

Thanks, fixed that. cue command uses even more resources Command being timed: "cue cmd -t debug -t name=test -t namespace=test -t mv=1.0.0 -t kv=1.28.0 build" User time (seconds): 53.65 System time (seconds): 0.59 Percent of CPU this job got: 152% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:35.55 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 1037592 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 263022 Voluntary context switches: 8109 Involuntary context switches: 1704 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

[stefanprodan]

Yeah I was expecting this, please open an issue in the cue lang repo here https://github.com/cue-lang/cue

[eshubin]

Do you think I can use the repo with timoni declaration for this bug report or better to extract something smaller?

[stefanprodan]

As long as they can reproduce it with just the cue command I think it’s fine to point them to the repo with the Timoni module, in the end a Timoni module is just a CUE module.

[eshubin]

CUE's master branch is rocket faster and uses 3 times less memory for my example https://github.com/eshubin/timoni-bug/commit/d405e410b987b1a18bcadf02f6f34ca5aa2003f5.

[eshubin]

Do you plan to integrate the newer version of CUE? CUE v0.12.1 does not have that optimization https://github.com/eshubin/timoni-bug/commit/23726cd700498d17fb5269726ba8437507e27d48

[stefanprodan]

If the optimization is in the latest stable CUE release we can update Timoni deps and make a release. I can do that next week.

[eshubin]

only in the master branch for now.

[stefanprodan]

@eshubin can you please run the tests with Timoni v0.25.0. I've updated CUE to v0.13 which should come with major performance improvements compared to the previous versions.