mullvadvpn-app icon indicating copy to clipboard operation
mullvadvpn-app copied to clipboard

High CPU use by configd process with macOS client connected

Open paulrudy opened this issue 2 years ago • 41 comments

Issue report

Operating system: MacOS Monterey 12.2

App version: 2022.1

Issue description

When the Mullvad client is connected, I see the configd process running at 100-110% CPU in Activity Monitor. When I disconnect, configd returns to 1-2% Leaving the client connected but changing settings for blocking ads, trackers, and malware has no effect.

paulrudy avatar Mar 05 '22 20:03 paulrudy

Issue seems to have disappeared after rebooting in Safe Mode (M1 Macbook Air)

paulrudy avatar Mar 06 '22 18:03 paulrudy

I spoke too soon. The issue has reoccurred.

paulrudy avatar Mar 06 '22 19:03 paulrudy

I can only notice some spikes on my own machine, Are you by any chance running some other software that is trying to configure your DNS configuration? Did this happen with a previous version too?

The daemon is attempting to listen for changes to your system's DNS configuration and reapply the configuration it needs constantly, and if there's another actor in the system trying to do the same thing, the high CPU load would probably be the end result. We have some plans of changing this behavior in some circumstances (when you'd be using resolves on your local network), but we've not come around to that just yet.

pinkisemils avatar Mar 07 '22 11:03 pinkisemils

Hi, thanks for checking. I don't have any software that configures DNS. I have iCloud Private Relay turned off. I can't recall that it was a problem before the current version, but I also don't think I checked before.

paulrudy avatar Mar 11 '22 18:03 paulrudy

Ok, there may be a connection between this behavior and something about iCloud Private Relay. When configd is using a lot of CPU, if I go to System Preferences > Apple ID > iCloud, the entry for Private Relay shows a spinning progress icon in place of the "Options" button, and no checkbox. It often doesn't stop spinning until I disable the Mullvad VPN. Once I disable the VPN, then Private Relay resolves to an unchecked checkbox and I see the "Options" button. So even though I have Private Relay disabled, perhaps something about MacOS is attempting to check a setting related to Private Relay, and can't complete it with the VPN enabled? Or perhaps there is no relationship.

I tried signing out of iCloud and signing in again. I also tried installing this configuration profile that disables Private Relay. The problem persisted anyway.

paulrudy avatar Mar 13 '22 00:03 paulrudy

Thanks, this is very valuable information as far as debugging this goes. It does seem like there's a fight going on between our daemon and some other piece of software, each trying to force a different DNS configuration. We will investigate what a good solution to this might be.

pinkisemils avatar Mar 14 '22 11:03 pinkisemils

For what it's worth I have exactly the same issue, including the iCloud settings entry for Private Relay showing a spinning progress icon in place of the "Options" button and no checkbox.

I am quite sure I don't have any macOS software specifically interfering with DNS however my router, an AmpliFi system, is configured to use Cloudflare's DNS so I wonder if this is causing an issue with the daemon, perhaps with macOS constantly hearing from the router that the DNS lookups should be to 1.1.1.1 and then the Mullvad daemon listening for changes and undoing it etc.

I'd be happy to try and help debug?

dgm34 avatar Mar 14 '22 11:03 dgm34

The router won't try to actively enforce a configuration, it will only be reapplied whenever your DHCP lease gets renewed. I have not gotten around to setting up a private relay myself just yet, but I suspect that it may be at fault here.

pinkisemils avatar Mar 14 '22 12:03 pinkisemils

I had this issue using Wireguard (home VPN, split tunneling for LAN only) over Mullvad (OpenVPN) on a network that doesn't support UDP (required by Wireguard).

I set my home server IP, which would be reachable through Wireguard in the above scenario, as DNS server in the Mullvad client.

In the end I resolved it by switching to Viscosity and setting DNS to "Disable"

ftruzzi avatar Mar 14 '22 15:03 ftruzzi

I'm having this issue as well. I am running the NextDNS client concurrently, but even after turning it off and quitting the NextDNS app, the Mullvad app still attempts to set something at a rate of 20 per second. I'm running Mullvad v 2022.1 on Big Sur.

*** Non-configd process (pid=74) attempting to set "Setup:/Network/Service/E97B9F93-EBB3-40B6-8674-4195A9E04A47/DNS" ***

mohamednazmi avatar Mar 15 '22 13:03 mohamednazmi

@mohamednazmi Is configd using a lot of CPU in this case where Mullvad is setting the DNS rapidly?

faern avatar Mar 15 '22 14:03 faern

@faern configd was using a lot of CPU, and in fact airportd was as well.

mohamednazmi avatar Mar 15 '22 14:03 mohamednazmi

(the most significant impact for me was on battery life, almost halved on M1 MBP!)

ftruzzi avatar Mar 15 '22 14:03 ftruzzi

@mohamednazmi can you try uninstalling NextDNS? It's still a bug, but it's important to deduce if it happens even if our app is the only one managing DNS on a given machine. Could you verify when this happens which PIDs belong to which applications from the logs? You should be able to use pidof $PID to see which application is setting the DNS config, and pidof can be installed via brew.

I've upgraded my machine to Monterey, and have tested out the private relay, and it seems to work well - the private relay disables itself when our app is installed, and can only be re-enabled if our app is uninstalled. This behavior is not ideal, ideally just not being connected via our app should be sufficient to use the private relay. However, I haven't been able to reproduce this issue.

Regardless of reproduction of the bug, I will look at debouncing the DNS config changes and detecting if there's a conflict going on if there's more than 10 DNS config changes in a given second and just error out the client. I can't imagine that DNS is actually working whilst our daemon is fighting with some other DNS configurator, so ending up in the error state will at least stop pegging the CPU.

pinkisemils avatar Mar 16 '22 11:03 pinkisemils

Apologies for interjecting but for the avoidance of doubt this issue happens on my machine even though I have never enabled private relay. So it's very weird I get the spinning progress icon in the private relay settings.

dgm34 avatar Mar 16 '22 11:03 dgm34

@pinkisemils I'll do that for the next 24 hours and report any anomalies, though I'm not expecting the CPU spikes anymore since they're not fighting over who gets to replace the DNS (for additional reference, between then and a few minutes ago I had restarted the machine and installed the Wireguard app to use concurrently with NextDNS, and the CPU has settled down).

In the initial report, the pid was definitely mullvad-daemon.

mohamednazmi avatar Mar 16 '22 11:03 mohamednazmi

@dgm34 Apologies if my previous message implied doubts about these reports. I don't doubt mullvad's client has issues here, I'm just trying to figure out the best way to reproduce the issue.

I do wonder, if this is something that only happens in 2022.1 or has this happened before, with earlier versions of our client?

pinkisemils avatar Mar 16 '22 12:03 pinkisemils

@pinkisemils No worries! And I think I can help with that - I reverted to the previous client (2021.6) a few days ago and it's totally fine. I think it's something introduced in 2022.1 👍

dgm34 avatar Mar 16 '22 12:03 dgm34

@pinkisemils it's been more than 24 hours with no CPU spikes.

mohamednazmi avatar Mar 19 '22 01:03 mohamednazmi

@mohamednazmi just to confirm, are you not getting any CPU spikes with the 2022.1 client?

pinkisemils avatar Mar 21 '22 10:03 pinkisemils

That's right, when running without the NextDNS client.

mohamednazmi avatar Mar 21 '22 10:03 mohamednazmi

I'm continuing to get high CPU with the configd process. I don't use NextDNS

paulrudy avatar Mar 21 '22 15:03 paulrudy

Has there been any progress on this? I use Big Sur, which does not offer iCloud Private Relay, and have never used NextDNS.

I've also confirmed that this issue does not occur when I instead use Viscosity with the Mullvad config files.

@pinkisemils Are there any commands that we can run to help diagnose this or otherwise provide you config information to help with reproducing the issue?

eytanhanig avatar Apr 15 '22 20:04 eytanhanig

@eytanhanig I can only suggest you try and see if there's any process that might be changing the DNS configuration, possibly by looking at the Console app, but these efforts have been futile before.

I have made some changes that should hopefully resolve the issue and in the worst case it will at least fix the CPU usage. I can produce a build with my changes if anyone here would be willing to test it out. Unfortunately I haven't been able to reproduce the issue. If the trhashing continues with my test build, I can at least log the conflicting configuration which can hint at what's thrashing here.

pinkisemils avatar Apr 19 '22 13:04 pinkisemils

For the more adventurous of you, there's a test build (and it's signature). It will definitely solve the issue with CPU usage, but it might also force the client to enter the error state - if this occurs, please do check the output of scutil --dns to get a hint as to what might be attempting to change the DNS config.

pinkisemils avatar Apr 20 '22 21:04 pinkisemils

The recently released beta also contains fixes for this bug, please do report back if this isn't fixed.

pinkisemils avatar May 17 '22 13:05 pinkisemils

I was having the high CPU utilization issues reported in this thread (but not the icloud spinning ball), so I grabbed the build above. "Unable to set system DNS server, please contact support." I'll send the output of scutil --dns to support.

ndw avatar May 24 '22 08:05 ndw

Are you running any other network-related software that might be trying to apply network settings? It doesn't seem like config that is being applied is special in any way.

pinkisemils avatar May 24 '22 11:05 pinkisemils

Emīls Piņķis @.***> writes:

Are you running any other network-related software that might be trying to apply network settings? It doesn't seem like config that is being applied is special in any way.

Not that I’m aware of. I’ll do some experiments. Disabling antivirus didn’t have any effect. I’ll try rebooting in a cleaner environment and seeing what I see.

                                    Be seeing you,
                                      norm

-- Norman Tovey-Walsh @.***> https://nwalsh.com/

He who seeks happiness for himself by making others unhappy is bound in the chains of hate and from those he cannot be free.--The Dhammapada

ndw avatar May 24 '22 14:05 ndw

I experience exactly the same behaviour as @ndw. I'm pretty confident I am running no other software that's messing with network settings.

dgm34 avatar May 24 '22 18:05 dgm34