Uninstall of a lot of modules takes very long time

[jasmin-j]

Hi!

My DKMS installs/uninstalls 617 modules. The install step is pretty fast (appr. 40..60 seconds), but the uninstalling takes 40 minutes. I tracked this down to this code in "do_uninstall":

         echo $"   - No original module was found for this module on this kernel."
         echo $"   - Use the dkms install command to reinstall any previous module version."

         # Remove modules_conf entries from /etc/modules.conf if remake_initrd is set or if this is last instance removed
         if [[ $remake_initrd ]] || (do_status $module $module_version | grep -q "installed"); then
             echo $""
             moduleconfig_remove "$1"
         fi

This gets executed for each "*.ko" module. It calls "do_status" and this calls "is_module_built" resulting in reading dkms.conf and checking if the modules exists in the file system. Is this part really required to be executed for each module in dkms.conf? The time required for this is a quadratic function of the installed modules.

I did a test with putting this code in front of the loop:

    if (do_status $module $module_version | grep -q "installed"); then
        was_installed="true"
    fi

And checking it after the loop:

    if [[ $remake_initrd && $was_installed ]]; then
        echo $""
        moduleconfig_remove "$1"
    fi

This worked on my Ubuntu machine really fast now (20..30 sec).

I suggest to modify "do_uninstall" either the way I did or to find another solution which doesn't traverse the whole modules for each uninstall module (quadratic effort).

BR, Jasmin

[scaronni]

Can you make a PR for this? BTW, out of curiosity, why do you have 617 modules to be built on your system? Are you backporting drivers from one kernel version to another?

[scaronni]

Nevermind, you are explaining it in the other issue.

[GoPerry]

If you have 617 modules. i think you can merge all the patches into your kernel will be better.

[GoPerry]

@jasmin-j

[jasmin-j]

If you have 617 modules. i think you can merge all the patches into your kernel will be better.

I want a DKMS because this shall be a package to be installed for all existing old Kernels. With the DKMS someone can add the newest media_tree drivers to an older distribution Kernel without fiddling with media_build.

[jasmin-j]

Can you make a PR for this?

I hope so in the next weeks. I am very busy and I need to test this again on my target platform.

[ali1234]

I tried moving the "installed" check outside the loop as suggested above and this cut uninstall time from over 1 hour to under 10 seconds.

Note that this problem, with an identical use case of building linux_media, was identified and apparently fixed before, 8 years ago, see:

https://bugs.launchpad.net/ubuntu/+source/dkms/+bug/786672

[ali1234]

Another thing I noticed is that module_status_built_extra calls diff on the module twice, once itself and once in find_actual_dest_module_location. Refactoring to only call diff once would globally reduce the overhead for do_status.

However, do_uninstall only calls do_status to check if the module is installed. Since the two diffs are only run if the module is installed, it seems like they are unnecessary for the purpose of do_uninstall and some other function should be used to determine install status instead.

All of this in addition to moving the check outside of the loop should give a nice speedup.

[AlexFolland]

This discussion seems to have come to a halt almost 4 years ago now, but it still seems relevant to me. Is there any progress available?

[Anonymous1157]

I have the exact same problem with the exact same media_build package and this is making it enfuriating to debug automation. It takes 60 seconds to install modules and nearly three hours to remove them.

What can I do to help get this fixed?

[evelikov]

Over the last couple of years we have done various rework, so I'm curious if the execution time is still as severe.

The analysis is useful although some input data for reproduction would be great. Without the latter personally I cannot justify spending too much time looking at this - sorry. Something like the following would be great:

• Fedora 99
• dkms version X, provided by the distro
• dkms modules A, B, C - A and B are provided via URL, while C comes from the distro
• kernels F G H are installed, distro provided
• the output of dkms status is ...
• running some command takes X minutes and produces some output

Alternatively a MR would be appreciated - one that includes a test or two, would be even better

[AlexFolland]

I did a test with putting this code in front of the loop: And checking it after the loop: This worked on my Ubuntu machine really fast now (20..30 sec).

I tried moving the "installed" check outside the loop as suggested above and this cut uninstall time from over 1 hour to under 10 seconds.

It sounds like this thread already has the solution and it just need to be implemented. If anyone has a dev environment for this and can move the "installed" check outside the loop in do_uninstall, I think many users would appreciate it. While I understand work on this is voluntary, I was surprised when I saw that this thread petered out after having multiple users claiming they fixed it locally with extremely dramatic improvements (from over 1 hour to under 10 seconds, for example).

[scaronni]

Well, as @evelikov said on the message before, merge requests are welcome.