After some recent changes, pw-top has a lot of ERR (also a lot of audio crackling)

[gabriele2000]

EasyEffects Version

7.0.4

What package are you using?

Flatpak (Flathub)

Distribution

Pop!_Os 22.04

Describe the bug

Copy-paste from https://github.com/pop-os/pop/issues/2881

Recently I updated pipewire, removed a tool called autocpu-freq and installed system76-scheduler. I also use EasyEffects for sound post-processing; it may be the culprit, it may be not, this issue happened since a week when EasyEffects was at version 7.0.3.

Version 7.0.2 was at least 2 weeks old so I don't thing it's the main issue, something else happened.

Expected Behavior

No audio crackling and not so many errors.

Debug Log

No response

Additional Information

• Power on your PC
• Open a slightly heavy game
• Open pw-top
• Listen to the crackling
• Open EasyEffects, even if the process was already running in background
• Now the crackling is less present, you may close it now, the problem will be at MAX intensity when you'll restart your PC
• Using pw-top, notice that with every crackling there is an error

[wwmm]

Based on the pw-top image the plugins are not the problem. The only ones known for causing crackling on latency changes are the convolver and the crystalizer. The ones you are using handle latency and sampling rate changes just fine. The latency used by PipeWire 2048 / 96000 = 0.021 seconds also should not be a problem. It is rare a system to have crackling at this range. But you may try to force a higher quantum to see if there is any change. PipeWire's wiki has some examples showing how to temporarily set a fixed quantum.

As far as I remember the errors in pw-top are mostly related to xrun. So something odd may be going on between PipeWire, ALSA and the soundcard. I am a little skeptical we can do something about this on EasyEffects.

[gabriele2000]

Based on the pw-top image the plugins are not the problem. The only ones known for causing crackling on latency changes are the convolver and the crystalizer. The ones you are using handle latency and sampling rate changes just fine. The latency used by PipeWire 2048 / 96000 = 0.021 seconds also should not be a problem. It is rare a system to have crackling at this range. But you may try to force a higher quantum to see if there is any change. PipeWire's wiki has some examples showing how to temporarily set a fixed quantum.

It was working just fine a week ago, even more. Some update or something else must've broke the perfectness. The kernel isn't an issue (I'm using xanmod 6.3.1).

The weird thing is that after opening EasyEffects the problem is "fixed" for a good 30%.

As far as I remember the errors in pw-top are mostly related to xrun. So something odd may be going on between PipeWire, ALSA and the soundcard. I am a little skeptical we can do something about this on EasyEffects.

Yeah that's why this is the copy of another issue. I'm skeptical and the only real possible issue is the package system76-scheduler, but I honestly don't know... it only changes priority of pipewire stuff to a higher level, so I guess it's not the culprit.

It would be weird if that package, made to optimize, causes such massive issues.

Also, my MSI laptop is advertised as "192kHz capable" officially, and it always handled it really well. The only reason I've lowered it is thanks to Guitarix.

One question: while I was still using Windows, 3 years ago, I had this tool for monitoring the system latency. Apparently my hardware is strange because it was telling me that I had a medium response time of 500us, which apparently is bad?

EDIT: The tool is called "LatencyMon"

[wwmm]

Apparently my hardware is strange because it was telling me that I had a medium response time of 500us, which apparently is bad?

I am not sure I understand the question. But based on what I have seen there are hardware that can't even use a latency of 20 milliseconds without crackling. My laptop for example. While my desktop can go close to 1 millisecond just fine. I do not know under what conditions this windows software was measuring latency but if it was able to go this low without causing crackling then your hardware can handle low latency just fine. Something in pipewire or alsa must have changed for you to be having problems now.

[gabriele2000]

I am not sure I understand the question.

That software was telling me that this response time was "high"... it's weird though, even if we forget about that software, my laptop is a gaming laptop, it would be weird for it not to sustain low latencies... Heh, the same laptop that has a loud-ass coil while if I enable "Intel Turbo-Boost" because of the sudden Voltage changes

[wwmm]

That software was telling me that this response time was "high"

Then it was measuring a different kind of latency and no audio latency. There is no way an audio latency of 500 microseconds is high.

[gabriele2000]

I have proof that somehow EasyEffects is the culprit. I'm encoding the video, then I'll upload to youtube and I'll send the link. The video is almost 2 minutes long.

You WILL hear the PAIN ahah! There you go

Of course this is an extreme case, it happens sometimes when I open OBS, but the issue is still the same.

[wwmm]

You WILL hear the PAIN ahah!

That noise is typical of the system not being able to handle the audio latency being used by the server. In the video PipeWire latency starts at 256 / 96000 = 0.0026 seconds. The server is using this value because games usually request unreasonable low audio latency. Some systems will handle this well others not. xrun is something tricky to fix. Just having a powerfull cpu is not enough. The audio server and its interaction with the card driver also matter.

When you set the fixed value the sampling rate was different but the latency was still very low 128 / 48000 = 0.0026. Same latency, same noise.

I have proof that somehow EasyEffects is the culprit.

PipeWire is the one manageing the filter's realtime thread as well as deciding the latency. What in turns sets the buffer size. The fact that having EasyEffects running makes any difference is because the stress on the audio server increases when there are plugins processing audio in its realtime thread.

I think that the real question in these cases is if there is something that can be changed in PipeWire configs that will make this crackling go away. If I remember well PipeWire has some configuration options that affect how it sets some ALSA parameters. They may help to fix this.

[gabriele2000]

The fact that having EasyEffects running makes any difference is because the stress on the audio server increases when there are plugins processing audio in its realtime thread.

The weird thing is that opening EasyEffects at least once every boot, fixes the problem for 30% of the intensity, which is weird, very weird...

PipeWire is the one manageing the filter's realtime thread as well as deciding the latency.

So, as I guessed days ago, Pipewire HAS an automated system somehow... the fact that I'll have to set manually a buffer size with /96000 for a resonable latency is baffling since that issue is completely random... I don't even know WHEN it started because I was so fucking lazy to document it...

[wwmm]

So, as I guessed days ago, Pipewire HAS an automated system somehow... the fact that I'll have to set manually a buffer size with /96000 for a resonable latency is baffling since that issue is completely random... I don't even know WHEN it started because I was so fucking lazy to document it...

Yes. PipeWire adjusts latency on the fly by default and if you edit its configuration files it can do the same with the sampling rate in other to avoid resampling. The dynamic latency procedure tries to satisfy the client requesting the lowest latency. Games for some reason ask for latency that is unnecessarily low in most situations. As a result Pipewire is using those values you saw.

There are ways to force a minimum quantum size (latency) in PipeWire configuration files. But it would be good to hear from them if there is a way to make the noise go away without having to force higher latency values.

[gabriele2000]

There are ways to force a minimum quantum size (latency) in PipeWire configuration files. But it would be good to hear from them if there is a way to make the noise go away without having to force higher latency values.

I'll link the issue to my future Pipewire's gitlab issue... for now I believe that export PIPEWIRE_LATENCY="8192/96000" will be safe enough until the problem is fixed as it was more than a week ago.

[wwmm]

for now I believe that export PIPEWIRE_LATENCY="8192/96000" will be safe enough until the problem is fixed as it was more than a week ago.

When I am playing I usually do a "per game solution" by exporting PULSE_LATENCY_MSEC before launching them. In my opinion it is better than forcing higher latency all the time.

[gabriele2000]

PULSE_LATENCY_MSEC

I'll do it this way then.

I usually do a "per game solution"

I'd do absolutely nothing but now every game has crackling one way or another... I honestly believe that something broke in Pipewire.

• It's not the kernel (reverted, not fixed)
• It's not the scheduler (purging it won't fix a thing)
• With my luck Pipewire's team will say "Works for me" or "Won't fix, invalid" but I'll try

Issue linked to Pipewire's issue tracker https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/3198

[violetmage]

Shower thoughts:

@gabriele2000 have you tried monitoring / logging your cpu frequency when the xruns start occurring?

[gabriele2000]

Shower thoughts:

@gabriele2000 have you tried monitoring / logging your cpu frequency when the xruns start occurring?

HMMMM... it's a good idea, let's see...

UPDATE: Nah, it's still to the maximum. I excluded possible throttlings since the beginning: I disabled PROCHOT from the BIOS (Processor Hot temperature response, something like that) and undervolted my CPU since years.

[wwmm]

I wonder if for some weird reason the plugin's thread managed by pipewire is not with realtime priority. The next time you run easyeffects find its pid number and execute chrt -ap pid_number. There should be one thread set to SCHED_RR.

[gabriele2000]

I wonder if for some weird reason the plugin's thread managed by pipewire is not with realtime priority. The next time you run easyeffects find its pid number and execute chrt -ap pid_number. There should be one thread set to SCHED_RR.

The priority is "Very High"

Since we're talking about scheduling tasks... I'll open a bug report for system76-scheduler package linking here and on my gitlab issue before going to bed.

[wwmm]

The priority is "Very High"

If that pid number is indeed from EasyEffects something is wrong. For example here on my computer I get

pid 2021's current scheduling policy: SCHED_RR|SCHED_RESET_ON_FORK

for one of the threads.

[gabriele2000]

If that pid number is indeed from EasyEffects something is wrong

Looking into system76-scheduler priority system, here are the values...

The issue is that easyeffects is under recording, which means that it's not realtime:

[wwmm]

The issue is that easyeffects is under recording, which means that it's not realtime:

I think it should not mess with EasyEffects at all. The reason is that you only want one thread to have realtime priorities in EasyEffects. The one where the filters audio data is processed. And the one managing this thread is PipeWire. By messing with EasyEffects priority this daemon will either undo what PipeWire did or do something even worse that is giving more priorities to gtk threads than they should have.

[gabriele2000]

The issue is that easyeffects is under recording, which means that it's not realtime:

I think it should not mess with EasyEffects at all. The reason is that you only want one thread to have realtime priorities in EasyEffects. The one where the filters audio data is processed. And the one managing this thread is PipeWire. By messing with EasyEffects priority this daemon will either undo what PipeWire did or do something even worse that is giving more priorities to gtk threads than they should have.

I think I'll put easyeffects in its ignore list then.

[wwmm]

I think I'll put easyeffects in its ignore list then.

It is a good idea. Now I am curious about what it is doing to PipeWire. Here on my computer this is what I get for the pipewire and pipewire-pulse processes

pid 1444's current scheduling policy: SCHED_OTHER|SCHED_RESET_ON_FORK
pid 1444's current scheduling priority: 0
pid 1458's current scheduling policy: SCHED_RR|SCHED_RESET_ON_FORK
pid 1458's current scheduling priority: 20

Hopefully they are still like this on your system.

[wwmm]

Here on Arch Linux wireplumber also has one of its threads set to SCHED_RR

pid 1446's current scheduling policy: SCHED_OTHER|SCHED_RESET_ON_FORK
pid 1446's current scheduling priority: 0
pid 1455's current scheduling policy: SCHED_RR|SCHED_RESET_ON_FORK
pid 1455's current scheduling priority: 20
pid 1459's current scheduling policy: SCHED_OTHER
pid 1459's current scheduling priority: 0
pid 1469's current scheduling policy: SCHED_OTHER
pid 1469's current scheduling priority: 0
pid 1471's current scheduling policy: SCHED_OTHER
pid 1471's current scheduling priority: 0
pid 1487's current scheduling policy: SCHED_OTHER
pid 1487's current scheduling priority: 0

Based on your image this daemon is doing to it the same it did to EasyEffects. I do not know why wireplumber needs realtime priority for one of its threads but I imagine they have a good reason for it.

[gabriele2000]

Now I am curious about what it is doing to PipeWire

pid 1470's current scheduling policy: SCHED_FIFO
pid 1470's current scheduling priority: 49
pid 1522's current scheduling policy: SCHED_FIFO
pid 1522's current scheduling priority: 49

and pipewire-pulse

pid 1475's current scheduling policy: SCHED_FIFO
pid 1475's current scheduling priority: 49
pid 1516's current scheduling policy: SCHED_FIFO
pid 1516's current scheduling priority: 49

Wireplumber:

pid 1474's current scheduling policy: SCHED_OTHER|SCHED_RESET_ON_FORK
pid 1474's current scheduling priority: 0
pid 1513's current scheduling policy: SCHED_RR|SCHED_RESET_ON_FORK
pid 1513's current scheduling priority: 20
pid 1517's current scheduling policy: SCHED_OTHER
pid 1517's current scheduling priority: 0
pid 1543's current scheduling policy: SCHED_OTHER
pid 1543's current scheduling priority: 0

Apparently I just need to exclude EasyEffects for now.

UPDATE: excluding things put them at niceness 9 so it's the opposite of what I wanted

[wwmm]

I never tried PipeWire with SCHED_FIFO in any of my computers. The only thing I found about it in Pipewire's page is https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/1271. There they say aAck does this. So it may not be a problem. But If I remember well some of the GPU driver threads are set SCHED_FIFO. I wonder if having the audio server also set to SCHED_FIFO while playing games can have some kind of negative impact. In any case I am just guessing.

[gabriele2000]

I wonder if having the audio server also set to SCHED_FIFO while playing games can have some kind of negative impact. In any case I am just guessing.

Well, I'm going to risk my life and ping @mmstick for this question. (sorry the the double ping, I messed up an edit)

EDIT: Forgot to add... games are in a separate category so they've a lower priority (nice -6) while easyeffects has a slightly higher priority (nice -9)

BREAKING NEWS system76-scheduler IS NOT the problem, I purged it and it didn't change almost anything... either EasyEffects has something wrong or Pipewire is exploding over-time

@wwmm is there a way to completely reset EasyEffects, just in case? Of course I will export my presets to re-add them later

[wwmm]

Forgot to add... games are in a separate category so they've a lower priority (nice -6) while easyeffects has a slightly higher priority (nice -9)

This is fine. I was thinking about process like gfx_0.0.0 that are related to the gpu and are usually set to SCHED_FIFO. If I am not mistaken processes that use SCHED_FIFO are usually supposed to yield once they finish what they have to do. It is different from the other scheduling types where the process is kicked out once its timeslice is over. I have no idea about how PipeWire actually behaves when using a fifo priority.

[wwmm]

is there a way to completely reset EasyEffects, just in case? Of course I will export my presets to re-add them later

You can reset it running easyeffects -r. This resets its gsettings database but does not delete what is inside ~/.config/easyeffects/. This folder has to be removed manually.

But the thing is that EasyEffects does not even try to set priorities for its threads. The one configuring the plugins thread to realtime priority is PipeWire. If that is not working is because something is broken in PipeWire.

[gabriele2000]

But the thing is that EasyEffects does not even try to set priorities for its threads. The one configuring the plugins thread to realtime priority is PipeWire. If that is not working is because something is broken in PipeWire.

Ok I ran a reset, I restarted the computer and now I'm going to test stuff.

[wwmm]

Ok I ran a reset, I restarted the computer and now I'm going to test stuff

For now I would focus on the lack of realtime priority in the plugins thread. Although realtime is not going to guarantee crackling/stutter is not going to occur not having it does not help. It is not normal to have normal priority for this thread. PipeWire is not being able to properly configure it for some reason.

[gabriele2000]

Ok I ran a reset, I restarted the computer and now I'm going to test stuff

For now I would focus on the lack of realtime priority in the plugins thread. Although realtime is not going to guarantee crackling/stutter is not going to occur not having it does not help. It is not normal to have normal priority for this thread. PipeWire is not being able to properly configure it for some reason.

After a brief testing with OBS recording: Perfection

Jokes aside, I might have to somehow completely reset Pipewire... after reinstalling the daemon the priority of easyeffects is again higher, without it it was "normal".