[Scale] Evaluate the ability for GUAC to handle scale of data

[lumjjb]

Evaluate the scalability of GUAC queries based on how much data we are expecting to ingest and be available within the next few years.

This issue should include:

• sizing of the problem and data set used
• design a set of experimental queries to best evaluate scale based on use cases
• generation of a representative data set
• running a set of experiments to evaluate the performance (execution and disk/memory)

[MinerYang]

Could you provide the proxy.log and UI log? thx

[AllForNothing]

@jorgemansillacriado Is there any special character in your password?

[jorgemansillacriado]

Could you provide the proxy.log and UI log? thx same user first logon harbor return 500 code and second logon return 200 code Nov 21 13:xx x.x.x.x. proxy[xxx]: x.x.x.x - "POST /c/login HTTP/1.1" 500 0 "https://x.x.x.x/account/sign-in?redirect_url=%2Fharbor%2Fprojects" "Mozilla/5.0 (Windows NT x.x.x.x; WOW64) AppleWebKit/xxxxx (KHTML, like Gecko) Chrome/x.x.x.x Safari/xxxxx" 1.489 1.489 . Nov 21 13:xx x.x.x.x proxy[xxx]: x.x.x.x - "POST /c/login HTTP/1.1" 200 0 "https://x.x.x.x/account/sign-in?redirect_url=%2Fharbor%2Fprojects" "Mozilla/5.0 (Windows NT xxx; WOW64) AppleWebKit/xxxx (KHTML, like Gecko) Chrome/x.x.x. Safari/xxxx" 0.628 0.627 .

[AllForNothing]

@stonezdj Can help to check on this, seems a back-end-related issue.

[stonezdj]

Could you please post the core.log ?

[jorgemansillacriado]

Could you please post the core.log ?

Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] the request url is /c/login Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] Handler crashed with error runtime error: index out of range [0] with length 0 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/runtime/panic.go:1038 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/runtime/panic.go:90 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/pkg/ldap/ldap.go:158 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/core/auth/ldap/ldap.go:66 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/core/auth/authenticator.go:155 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/core/controllers/base.go:98 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/reflect/value.go:543 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.292 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/reflect/value.go:339 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/vendor/github.com/astaxie/beego/router.go:853 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:52 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/security/security.go:75 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/security/security.go:62 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/artifactinfo/artifact_info.go:60 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/transaction/transaction.go:62 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/lib/orm/orm.go:131 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/transaction/transaction.go:71 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/notification/notification.go:31 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/orm/orm.go:53 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/csrf/csrf.go:48 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/vendor/github.com/gorilla/csrf/csrf.go:297 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/csrf/csrf.go:68 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/session/session.go:34 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/log/log.go:33 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/requestid/requestid.go:43 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.293 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.294 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/metric/metric.go:51 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.294 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.294 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/trace/trace.go:28 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.294 #033[1;35m[C]#033[0m [panic.go:1038] /usr/local/go/src/net/http/server.go:2046 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.294 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/mergeslash/mergeslash.go:17 Nov 21 13:09:40 core[27384]: 2022/11/21 12:09:40.294 #033[1;35m[C]#033[0m [panic.go:1038] /harbor/src/server/middleware/middleware.go:57

[stonezdj]

The auth mode is LDAP auth, it seems that the LDAP search returns an entry without any value. What is the LDAP configuration? Is your LDAP server an OpenLDAP or Active Directory? what is the configure value of LDAP Group Membership Attribute?

[stonezdj]

It seems this issue is fixed in v2.5 and 2.6, see https://github.com/goharbor/harbor/pull/15864 could you please try the latest version?

[jorgemansillacriado]

It seems this issue is fixed in v2.5 and 2.6, see #15864 could you please try the latest version?

ok we're going to upgrade Harbor and when we do upgrade I'll tell you

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.