Response time on "cf apps" call on v2 vs. v3 api

[MMisoch]

Issue

We measured the response time on cf apps calls of the v2 api and the v3 api and observed a significantly increased response time on the v3 api.

Context

On one of our large landscapes we have a space with ~700 apps. If cf apps gets called it takes about 4-5 minutes which is barely acceptable for customers. However if we use cf7 apps the response time increases to ~2 hours (Yes we triple checked this) which is absolutely not acceptable. We dug deeper into this and found the following:

In the V2 api it is just called:

• /v2/spaces/someguid/summary - This call alone takes 300-450 seconds to come back. In the space with 700 apps it made about 111000 select statements during this time period. So the time increase per app on that scale seems to be ~0.5 - 1 second.

In the V3 api there are several calls made: First to get all the apps:

• /v3/apps?space_guids=someguid - This call takes about 5 seconds (totally fine)

The Problem seems to be that after the initial call that gets all apps, for every app found there are more calls:

• /v3/apps/someguid/routes - This call takes about 4-6 seconds
• /v3/apps/someguid/processes - This call takes about 1-2 second
• /v3/processes/someguid/sidecars - This call takes about 1-2 second
• /v3/processes/someguid/stats - This call takes about 1-2 second ... that are the ones that took the longest.

That adds up to a total time of ~700 apps x ~10 seconds = ~7000 seconds = ~2 hours.

So the time increase per app on that scale seems to be 10 seconds.

Steps to Reproduce

Not sure if reproduce-able due to the size of the landscape this problem was observed on. We were not able to reproduce the exact times mentioned here on a development landscape. Nevertheless we observed, that the time increase and the number of calls that are executed for cf apps and cf7 apps scale with the number of apps per space.

Expected result

300-450 seconds is barely acceptable for customers working with the cli and websites which use the cli to display all apps in a space are not usable at all. But the v2 (cf apps) can still be used with a bit of patience. However the v3 api (cf7 apps) with more than 2 hours of loading time is not useable with the mentioned amount of apps in a space. We would expect the loading time of the v3 api to be at least as fast as the v2 api.

Current result

The v2 api outperforms the v3 api on a big landscape that has a space with ~700 apps in it on the cf apps (cf7 apps) call.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172892568

The labels on this github issue will be updated when the story is started.

[reidmit]

Thanks for filing this! We have also noticed that cf7 apps is quite slow, and we very recently did some work to investigate and fix it. We were able to eliminate lots of unnecessary requests and merge some per-app requests together. Here are some stories that we recently completed:

https://www.pivotaltracker.com/story/show/172461503 https://www.pivotaltracker.com/story/show/172625875 https://www.pivotaltracker.com/story/show/172625783

These changes are not yet widely available. They require the newest CAPI version (which may not be in a cf-deployment version yet), and the master branch of the CLI. You're welcome to try it out, if you want, or you can wait for these changes to be released.

I do not think we have matched the performance of v2/v6 exactly, but we made some dramatic improvements!

Some rough numbers from our testing on a small BOSH-lite environment with 20 apps in the targeted space:

• cf6 took 1.414s
• cf7 before our optimizations took 9.609s (this is the version you're using, I believe)
• cf7 after our optimizations took 3.265s

There is another story, not yet prioritized, which may let us improve the performance even more. This would involve inlining the process stats data on the /v3/processes resources, eliminating the need for additional requests for each process. We're not yet sure how much this would help, or if this is an approach we want to take: https://www.pivotaltracker.com/story/show/172625808