flagd idle memory consumption 40mb (higher than expected)

[gnesher]

We've run some initial tests on flagd memory consumption, single file system syncer with < 10 flags with no reads and see ~40mb consumption.

This seems high for a Go application, and I wanted to know if this is in line with what we should expect or if there's anything we can do to reduce the footprint.

The goal is to support relatively small instances where 40mb of memory does make an impact.

[tangenti]

Hi @gnesher ,

Could you provide more details of your test? What version of flagd did you use? Did you run the flagd with or without container? How did you measure the memory usage of flagd?

[gnesher]

@tangenti We are using flagd: v0.11.4 (from May we can try to upgrade, but I've not seen any memory improvments and didn't want to change version during our tests)

We are installing flagd locally (no container) and using the rss field in 'ps' would be the simplest way to get this value (we use a slightly more complex way of monitoring the entire setup, but I verified that the results are the same)

i.e. ps -o rss= -p <proccess_id>

[m-olko]

Hi @gnesher ,

I did some investigation and wanted to share my thoughts and findings.

When ran newly built flagd (current main so ~v0.12.9) the rss value on my computer was even higher: ~65 MB with only two flags. I wanted to take a closer look on memory metrics from go, so I utilized pprof built into flagd and available after building flagd with: go build -a -ldflags "-X main.version=${VERSION} -X main.commit=${COMMIT} -X main.date=${DATE}" -o ./bin/flagd ./flagd/main.go ./flagd/profiler.go The flagd instance then provides service on port 6060 with the go pprof package

Looking at the heap size (which should be the biggest part of used memory) shows that that the flagd really uses ~14MB of data: # HeapInuse = 14082048 When forced to run Garbage Collector, the usage drops even more to under 12MB: # HeapInuse = 11501568 And enforcing soft memory limit with export GOMEMLIMIT=10MiB allows us to reduce footprint to under 10MB: # HeapInuse = 9969664 All this data is taken from http://localhost:6060/debug/pprof/heap?debug=1 endpoint of pprof.

Quick internet research shows possibility that the memory, even though released by the Go, is not updated in the system - resulting in increased rss value: https://github.com/golang/go/issues/39779#issuecomment-648253607

So it is possible that the flagd would run without problems with much less resources available. Have you had issues with running your instance on machine with limited resources or limited by the cgroup? Or are your findings purely theoretical for now?

[m-olko]

Also, following up on the resource used, I decided to investigate what is using such amount of data. I knew that sometimes libraries related to cloud resources like to download full configuration files, and pure initialization of them might use significant amount of memory. To test that i commented out all the sync types besides the file one, to remove the footprint of gcp, aws, azure, kubernetes etc related libraries. You can see updated code on my fork https://github.com/m-olko/flagd/tree/memory_investigation

The rss value of such stripped flagd decreased to ~25MB, while the heap size to under 3MB.

Maybe it would be beneficial to create some compile-time flags that will allow for removal of mentioned libraries when used memory needs to be minimal?

@toddbaert maybe you could share your opinion on such feature?

[gnesher]

@m-olko thanks, I've also forwarded your response to our performance team and will see what they say

Our software is installed on customer instances (or local machines). We've seen a rise in the use of low-end setups, where memory/CPU consumption can hurt customers, so we are tracking any increase in our setups.