[Bug][config-ui] Edit Data Scope and Scope Config fails to load with too many scopes

[gustavobini]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

I have a Devlake project with a Jenkins connection configured with 91 data scopes.

When I click on Edit Data Scope and Scope Config, navigating to https:///projects//<connection_id>, the page never stops showing the loading screen.

Upon inspecting the network requests, the page sends a total of 95 requests at once. The response times are from a couple of milliseconds to a minute (timed out by the load balancer). Most of the requests fail with a 500 status code.

On the API logs, there are several connection errors to the database.

time="2024-04-19 10:31:24" level=info msg="\x1b[31;1m/app/impls/dalgorm/dalgorm.go:215 \x1b[35;1mfailed to connect to `host=localhost user=redacted database=devlake`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)\n\x1b[0m\x1b[33m[30169.975ms] \x1b[34;1m[rows:0]\x1b[0m SELECT * FROM \"_tool_jenkins_jobs\" WHERE connection_id = '1'  AND full_name = 'redacted'  ORDER BY \"_tool_jenkins_jobs\".\"connection_id\" LIMIT 1"
time="2024-04-19 10:31:24" level=error msg="HTTP 500 error\n\tcaused by: failed to connect to `host=localhost user=redacted database=devlake`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address) (500)\n\tWraps: (2) failed to connect to `host=localhost user=redacted database=devlake`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)\n\tWraps: (3) dial tcp [::1]:5432\n\tWraps: (4) connect\n\tWraps: (5) cannot assign requested address\n\tError types: (1) *hintdetail.withDetail (2) *pgconn.connectError (3) *net.OpError (4) *os.SyscallError (5) syscall.Errno"

Some notes:

• The number of database connections is far from the limit
• There are a lot of free compute resources in both the database and the API

What do you expect to happen

I expected the page to load the data.

How to reproduce

1. Create a connection
2. Add at least 90 data scopes to this connection
3. Create a project
4. Link the connection to the project
5. Load the project's page to edit data scope and scope config

Anything else

I initially noticed this problem with https://github.com/apache/incubator-devlake/releases/tag/v1.0.0-beta5 as I wanted to use the Jenkins multi-branch workflows, but even after reverting to https://github.com/apache/incubator-devlake/releases/tag/v1.0.0-beta1 the problem persists.

Version

v1.0.0-beta5

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[talfirevic]

Same behavior, same version here. Number of scopes is 1100+

[d4x1]

In my local env with the main branch's codes with MySQL 8.x,I created a GitHub conenction with 100+ scopes,

and I can open the connection details successfully with limited time.

There is an issue that scope's details are fetched seperately, it will create 100+ requests at the same time, which I think could be improved.

As to the errors "cannot assign requested address" in your backend log, considering you are using docker, I think it's some limilations in docker, can you have a try at this https://stackoverflow.com/questions/66233698/cannot-assign-requested-address-localhostxxxx-docker-linux-containers ?

@talfirevic what's your environment ?

[gustavobini]

As to the errors "cannot assign requested address" in your backend log, considering you are using docker, I think it's some limilations in docker, can you have a try at this https://stackoverflow.com/questions/66233698/cannot-assign-requested-address-localhostxxxx-docker-linux-containers ?

@d4x1 this is not running in Docker. It's in a GKE cluster and the database request goes through a cloud-sql-proxy sidecar. However, I will also look into this once I'm back from work vacation.

There is an issue that scope's details are fetched seperately, it will create 100+ requests at the same time, which I think could be improved.

Yes, this is the solution I was looking for from opening this issue. From a brief look, this seems to be handled by the code framework though, so doesn't seem that simple.

[gustavobini]

@d4x1 I fixed this by increasing the resources of the cloud-sql-proxy sidecar container. However, I still think it's valid to improve the way data is fetched.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.