pulp
Large memory spike at the very end of repository sync
Version "rpm": "3.23.0", "core": "3.38.0"
Describe the bug We recently set our memory limits to 4G on our pulp worker pods, and this was not enough to sync RHEL 8 Appstream. The sync would be killed, but all the steps would be complete.
I did a local sync and saw memory spike at the very end of the sync up to 3.7 GB. Here's a link to the memory stats: https://docs.google.com/spreadsheets/d/1AE3_QKFdQIO6MUSUgWFMm_-fDEzUI6IYurp3-4AE-fA/edit#gid=0
Its very likely that it spiked over 4GB, but due to sampling frequency, i only saw 3.7.
To Reproduce Sync https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os (with certs that let you access it)
Expected behavior Not a giant memory spike if possible?
Additional context Here's the full task output when its killed please excuse the funky formatting) :
{
pulp_href: "/pulp/ae86a858/api/v3/tasks/018bb161-8629-749b-ae7a-ce6b3a754c28/"
pulp_created: "2023-11-09T00:00:40.492087Z"
state: "failed"
name: "pulp_rpm.app.tasks.synchronizing.synchronize"
logging_cid: "3d30b27738fb479190dbaca9f8811a94"
started_at: "2023-11-09T00:28:48.278736Z"
finished_at: "2023-11-09T01:31:02.102107Z"
error: {
reason: "Killed by signal 9."
}
worker: "/pulp/default/api/v3/workers/018b9689-c1cb-7034-8e91-d4c4866b87ec/"
progress_reports: [
0: {
message: "Parsed Comps"
code: "sync.parsing.comps"
state: "completed"
total: 68
done: 68
suffix: null
}
1: {
message: "Downloading Metadata Files"
code: "sync.downloading.metadata"
state: "completed"
done: 7
suffix: null
}
2: {
message: "Downloading Artifacts"
code: "sync.downloading.artifacts"
state: "completed"
done: 7
suffix: null
}
3: {
message: "Associating Content"
code: "associating.content"
state: "completed"
done: 36908
suffix: null
}
4: {
message: "Parsed Modulemd"
code: "sync.parsing.modulemds"
state: "completed"
total: 717
done: 717
suffix: null
}
5: {
message: "Parsed Modulemd-defaults"
code: "sync.parsing.modulemd_defaults"
state: "completed"
total: 49
done: 49
suffix: null
}
6: {
message: "Parsed Modulemd Obsolete"
code: "sync.parsing.modulemd_obsoletes"
state: "completed"
total: 0
done: 0
suffix: null
}
7: {
message: "Skipping Packages"
code: "sync.skipped.packages"
state: "completed"
total: 6
done: 6
suffix: null
}
8: {
message: "Parsed Packages"
code: "sync.parsing.packages"
state: "completed"
total: 32880
done: 32880
suffix: null
}
9: {
message: "Un-Associating Content"
code: "unassociating.content"
state: "completed"
done: 0
suffix: null
}
10: {
message: "Parsed Advisories"
code: "sync.parsing.advisories"
state: "completed"
total: 3199
done: 3199
suffix: null
}
]
reserved_resources_record: [
0: "/pulp/ae86a858/api/v3/repositories/rpm/rpm/018b1a52-1006-746f-b465-0aa3ad9840a8/"
1: "shared:/pulp/ae86a858/api/v3/remotes/rpm/rpm/018b1a51-fc00-7f3e-9819-d94696313d83/"
2: "shared:/pulp/default/api/v3/domains/018b1a51-cfd6-7016-8872-6bc88173f27b/"
]
}
I've seen this before, it happens only when doing a complete-mirror mode sync, so I presume it's related to creating the publication.
we're actually not using mirror_mode. here's the completed output of the task, and you can see no publication was created:
{
"pulp_href": "/pulp/75d8fd70/api/v3/tasks/018bce17-9831-7cb1-adc8-3566f00666b0/",
"pulp_created": "2023-11-14T13:48:54.707083Z",
"state": "completed",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"logging_cid": "a53ebb89bfb242f299d72f702b583e48",
"created_by": "/pulp/75d8fd70/api/v3/users/1/",
"started_at": "2023-11-14T13:48:54.771681Z",
"finished_at": "2023-11-14T13:54:29.556635Z",
"error": null,
"worker": "/pulp/default/api/v3/workers/018bce15-cf48-7bac-9aaf-052b63754017/",
"parent_task": null,
"child_tasks": [],
"task_group": null,
"progress_reports": [
{
"message": "Downloading Metadata Files",
"code": "sync.downloading.metadata",
"state": "completed",
"total": null,
"done": 7,
"suffix": null
},
{
"message": "Parsed Modulemd",
"code": "sync.parsing.modulemds",
"state": "completed",
"total": 736,
"done": 736,
"suffix": null
},
{
"message": "Parsed Modulemd-defaults",
"code": "sync.parsing.modulemd_defaults",
"state": "completed",
"total": 49,
"done": 49,
"suffix": null
},
{
"message": "Parsed Modulemd Obsolete",
"code": "sync.parsing.modulemd_obsoletes",
"state": "completed",
"total": 0,
"done": 0,
"suffix": null
},
{
"message": "Skipping Packages",
"code": "sync.skipped.packages",
"state": "completed",
"total": 0,
"done": 0,
"suffix": null
},
{
"message": "Parsed Advisories",
"code": "sync.parsing.advisories",
"state": "completed",
"total": 3392,
"done": 3392,
"suffix": null
},
{
"message": "Downloading Artifacts",
"code": "sync.downloading.artifacts",
"state": "completed",
"total": null,
"done": 1,
"suffix": null
},
{
"message": "Un-Associating Content",
"code": "unassociating.content",
"state": "completed",
"total": null,
"done": 0,
"suffix": null
},
{
"message": "Associating Content",
"code": "associating.content",
"state": "completed",
"total": null,
"done": 39056,
"suffix": null
},
{
"message": "Parsed Packages",
"code": "sync.parsing.packages",
"state": "completed",
"total": 34810,
"done": 34810,
"suffix": null
},
{
"message": "Parsed Comps",
"code": "sync.parsing.comps",
"state": "completed",
"total": 68,
"done": 68,
"suffix": null
}
],
"created_resources": [
"/pulp/75d8fd70/api/v3/repositories/rpm/rpm/018bce17-9696-7216-bbab-3c708304eaf1/versions/1/"
],
"reserved_resources_record": [
"/pulp/75d8fd70/api/v3/repositories/rpm/rpm/018bce17-9696-7216-bbab-3c708304eaf1/",
"shared:/pulp/75d8fd70/api/v3/remotes/rpm/rpm/018bce17-9392-7bc8-b772-e6de14e0ee20/",
"shared:/pulp/default/api/v3/domains/018bce17-907e-71ba-a255-965ae2145f5a/"
]
}
let me grab the remote configuration
It's happening with the https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/ too
