guac
guac copied to clipboard
[ENT bug] Query hits PostgreSQL 65535 parameters limit
Describe the bug After a large dataset is ingested, the OSV certifier and others that queries for all packages hit the PostgreSQL 65535 parameters limit:
go run ./cmd/guacone certifier osv
{"level":"info","ts":1713447858.50912,"caller":"logging/logger.go:75","msg":"Logging at info level"}
{"level":"info","ts":1713447858.509234,"caller":"cli/init.go:69","msg":"Using config file: /Users/parth/Documents/pxp928/artifact-ff/guac.yaml"}
{"level":"error","ts":1713447861.422855,"caller":"cmd/osv.go:169","msg":"certifier ended with error: failed sources query: input: packages pq: got 563996 parameters but PostgreSQL only supports 65535 parameters\n","stacktrace":"github.com/guacsec/guac/cmd/guacone/cmd.init.func10.3\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacone/cmd/osv.go:169\ngithub.com/guacsec/guac/pkg/certifier/certify.Certify.func1\n\t/Users/parth/Documents/pxp928/artifact-ff/pkg/certifier/certify/certify.go:81\ngithub.com/guacsec/guac/pkg/certifier/certify.Certify\n\t/Users/parth/Documents/pxp928/artifact-ff/pkg/certifier/certify/certify.go:99\ngithub.com/guacsec/guac/cmd/guacone/cmd.init.func10.4\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacone/cmd/osv.go:180"}
{"level":"info","ts":1713447861.4229681,"caller":"cmd/osv.go:192","msg":"All certifiers completed"}
This happens based on how the query for packages is setup where it eager loads the packageName edges to reconstruct model.Package as shown below (WithName does the eager loading):
https://github.com/guacsec/guac/blob/358205b583a21f3ba196073c8b08521c6389ace4/pkg/assembler/backends/ent/backend/package.go#L50-L57
This similar issue may be faced by other nouns/verbs that do some type of eager loading or filtering.
To overcome this, we need to implement pagination in ENT: https://entgo.io/docs/tutorial-todo-gql-paginate/#add-pagination-support-for-query. More information found in the issue https://github.com/guacsec/guac/issues/1525
To Reproduce Ingest a large amount of SBOM and try to query without a filter. This will result in:
{
"errors": [
{
"message": "HasSBOM: pq: got 563996 parameters but PostgreSQL only supports 65535 parameters",
"path": [
"HasSBOM"
]
}
],
"data": null
}
Expected behavior Return data back with pagination
GUAC version main