argoproj
Server-side pagination to make ArgoCD UI a rocket
Summary
We're on ArgoCD 2.4.2+c6d0c8b with more than 2000 Applications and suffering from an annoying issue with slow UI loading default /applications endpoint. JFY: Gzip 9 is enabled on ArgoCD side and also on the Ingress level.
Depending on the user's internet connection and location UI loading can take 10s - 20s that's really slow and makes us think about the doubtful workaround, like redirecting users from /application endpoint to the custom one with a dummy Application.
Proposal
Endpoint /application returns one huge JSON that represents a set of ALL Applications managed by ArgoCD with all values, metadata, statuses, annotations, and the same for all managed k8s resources. It requires quite a lot of work on the client side and also on the server side for gzipping a big bunch of data.
Per discussion with @crenshaw-dev in the #argo-cd Slack channel, he mentioned that server-side pagination is one of the solutions that might be implemented to reduce data downloading on the client-side and increase UI performance significantly.
It would be great if the pagination will be customized, but it depends on implementation.
I guess it should be a really awaited feature due to there are a lot of issues/discussions about slow UI that were already reported/started. Example - https://github.com/argoproj/argo-cd/discussions/8446 where @rumstead already collected related issues.
I hope this is possible and really appreciate your time. Thanks!
Thanks for logging this one, I have subscribed to the post and will follow along.
I am not at that scale yet, but researching how well ArgoCD handles these kinds of numbers for when that time comes. Beyond the slow UI loading have you found it handles 2000+ without too many issues? What sort of system resources does your instance use?
Server-side pagination was implemented by Akuity (see their blog post https://akuity.io/blog/akuity-july-2023-update/). When we asked on Slack, we were told that the team at Akuity was planning on upstreaming it in around a month (see discussion here.
That's great news, really! Currently with 4K applications, the UI (/applications) is clearly no longer usable.
@alexmt's original estimate would put this in time for 2.9, so setting it for that milestone for now. :-)
@Whisper40 I haven't seen the code, but my guess is that it was fairly involved. I kind of doubt we'll be able to backport, but will certainly consider it.
In good news, 2.8 seems to be pretty solid, and I expect 2.9 to be a relatively light release. So hopefully the upgrade path will be painless.
@crenshaw-dev @alexymantha Hi, it seems that the 2.9 release has been published but i don't see any mention about pagination. Is it normal ? (Oh it seems it's for 2.10 :( )
@Whisper40 I don't think this has been announced to be included in any particular release yet.
It looks like it is going to be released in v2.10 https://github.com/orgs/argoproj/projects/25/views/14
The Akuity folks have told me they plan to get this into 2.10, so I've added the ticket to that column in the project. It's not a promise, just an indication of the current plan. :-)
I'd say, if they won't merge their patch soon it can be someone else with a completely different implementation and then Akuity would have a hard time to maintain their fork.
Sorry, for the delay with the PR. We ran into several issues along the way and had to take a step back to ensure good quality. As a short summary:
- CLI changes were absent. Akuity doesn’t provide customized Argo CD CLI to customers, so no related changes were implemented.
- Pagination is leveraging direct DB query to the etcd database. This is possible only if K8S is using KINE with DB as a backend. So this part has to be reworked as well.
Since the implementation is more involved than expected, I’ve started with a proposal that documents required changes: https://github.com/argoproj/argo-cd/pull/17222
Hello ArgoCD Team, @alexmt . What's the latest on when we think this feature will be released? Thanks.
I found out that during the list application phase, we're getting back a bunch of data we don't really need, like operationState and resources. Since we're gonna fetch these again from the server once someone clicks on a specific application, how about we trim down the data during the list application phase? Could this also speed up the page response time? I'm tweaking and testing stuff locally right now.
updates 2024.07.02: i add a function to prune application contents, and it looks like working well, In my ArgoCD test environment (v2.9.0), the original API response returned a JSON with 15.5k lines and a size of 616KB. After modifications, the JSON now has 4.45k lines and a size of 128KB. This represents a 71.3% reduction in the number of lines and a 79.2% decrease in data volume.
update 2024.12.25: add zstd support on argocd 2.11.5, in my prod enviroment, list all applications cost 46.8mb, and now it's only 384kb! reduced 99.2%.
func pruneApplication(a *appv1.Application) appv1.Application {
a = a.DeepCopy()
defer func() {
if e := recover(); e != nil {
log.Errorf("[edenz] error: %v\n%s", e, debug.Stack())
}
}()
if a.Status.OperationState != nil && a.Status.OperationState.Operation.Sync != nil && len(a.Status.OperationState.Operation.Sync.Resources) > 0 {
a.Status.OperationState.Operation.Sync.Resources = nil
fmt.Println("[edenz] Pruning OperationState", a.Status.OperationState.Operation.Sync.Resources)
}
if a.Status.OperationState != nil && a.Status.OperationState.SyncResult != nil && len(a.Status.OperationState.SyncResult.Resources) > 0 {
a.Status.OperationState.SyncResult.Resources = nil
fmt.Println("[edenz] Pruning SyncResult", a.Status.OperationState.SyncResult.Resources)
}
if len(a.Status.Resources) > 0 {
a.Status.Resources = nil
fmt.Printf("[edenz] Pruning Resources: %v\n", a.Status.Resources)
}
if len(a.Status.History) > 0 {
a.Status.History = nil
fmt.Printf("[edenz] Pruning History: %v\n", a.Status.History)
}
a.Annotations = nil
fmt.Println("[edenz] Pruning Annotations")
return *a
}
This is much awaited feature we are waiting for. As we are maintaining 4000 apps and the UI becomes slow and sometimes it is dead slow.
i add a function to prune application contents, and it looks like working well, In my ArgoCD test environment (v2.9.0), the original API response returned a JSON with 15.5k lines and a size of 616KB. After modifications, the JSON now has 4.45k lines and a size of 128KB. This represents a 71.3% reduction in the number of lines and a 79.2% decrease in data volume.
@ezxfv to prioritize that change, I think we'd need reason to believe that payload is a significant performance bottleneck. For now, it seems like CPU is the more significant bottleneck.
@crenshaw-dev yeah TTFB (Time to first byte) should be a metric of interest.
@crenshaw-dev Indeed, this method of reducing irrelevant information can only temporarily alleviate my current predicament, allowing the Argo CD homepage to barely load all applications (8000+). Upon a cursory examination of the performance on Chrome, the primary time consumers are script execution (9s) and rendering (5s). Would leveraging Web Workers and WASM contribute to accelerating this process? It appears that pagination remains the least affected solution by the growth in the number of applications.
@ezxfv I imagine there are a lot of things that could be tuned on the frontend to speed things up. But even if you manage to half the script execution/rendering time, you're probably not going to get as much a win as you would be cutting the number of processed applications by 90%.
There are a lot of ways we could improve performance. Pagination seems like the highest payoff for the least effort / maintenance burden.
Any news on this?
The proposal Alex wrote is now merged: https://github.com/argoproj/argo-cd/pull/17222
Implementation is pending.
Any update on this ? Maintaining a lot of apps (above 3-4k) start to feel painful with the UI I think having argo cd ui running on things like Nextjs with a full server side first approach would be a good thing But this is probably bigger workload too
I made the following two small changes: by delaying the rendering of non-essential components, the entire page now loads much faster, and memory usage has been significantly reduced. I have 9500 applications, and now the homepage opens in just 1-2 seconds, with memory usage significantly reduced. Search, filtering, and other functions have become incredibly smooth. Previously, it took over 20 seconds, and the page could even crash. In addition to this, there were some other minor changes, but these are the main improvements.
