modulemd-tools icon indicating copy to clipboard operation
modulemd-tools copied to clipboard

I/O intensive walk by createrepo_mod

Open praiskup opened this issue 4 years ago • 1 comments

The code eats in order of magnitude more IO than the optimized createrepo_c run on large repositories.

In copr, we run: /usr/bin/createrepo_mod /var/lib/copr/public_html/results/iucar/cran/fedora-31-x86_64 --database --ignore-lock --local-sqlite --cachedir /tmp/ --workers 8 --recycle-pkglist --update --skip-stat --pkglist /var/lib/copr/public_html/results/iucar/cran/fedora-31-x86_64/.copr-createrepo-pkglist

praiskup avatar Nov 24 '20 10:11 praiskup

Previous Copr code (before calling createrepo_mod) was just expecting the modulemd file in the root repo directory:

-def add_modular_metadata(opts):
-    if not os.path.exists(os.path.join(opts.directory, "modules.yaml")):
-        return
-    run_cmd(
-        ["/usr/bin/modifyrepo_c",
-         "--mdtype", "modules",
-         "--compress-type", "gz",
-         os.path.join(opts.directory, 'modules.yaml'),
-         os.path.join(opts.directory, 'repodata')], opts)

praiskup avatar Nov 24 '20 10:11 praiskup