consumer.setPreferredLayer(spatial: 1) makes mediasoup select layer 2 if layer 1 doesn't exist

[ibc]

Bug Report

Your environment

• mediasoup version: 3.14.16

Issue description

If the Producer has 3 encodings (so 3 simulcast spatial layers 0,1,2) and the encoding in the middle (encoding 1) has active: false, then when the consumer calls setPreferredLayers({ spatial: 1 }) mediasoup selects layer 2 instead of 0.

This can be tested by doing this in the demo app:

diff --git a/app/lib/RoomClient.js b/app/lib/RoomClient.js
index 85794c0..ce20ffa 100644
--- a/app/lib/RoomClient.js
+++ b/app/lib/RoomClient.js
@@ -1096,7 +1096,8 @@ export default class RoomClient
 							{
 								scaleResolutionDownBy : 2,
 								maxBitrate            : 1000000,
-								scalabilityMode       : this._webcamScalabilityMode || 'L1T3'
+								scalabilityMode       : this._webcamScalabilityMode || 'L1T3',
+								active                : false
 							}
 						);
 					}

[ibc]

What I see in SimulcastConsumer.cpp custom logs when stream 1 (middle one) is disabled and I pass from preferred layers 0:2 to 1:0:

   mediasoup:ERROR:Channel [pid:23746] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- testing spatial layer 0 +4s
  mediasoup:ERROR:Channel [pid:23746] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- testing spatial layer 1 +0ms
  mediasoup:ERROR:Channel [pid:28803] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- producerScore == 0, next undefined +1ms
  mediasoup:ERROR:Channel [pid:23746] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- testing spatial layer 2 +0ms
  mediasoup:ERROR:Channel [pid:23746] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- this is the preferred or higher spatial layer take it and exit +0ms
  mediasoup:ERROR:Channel [pid:23746] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- newTargetTemporalLayer = 0; +0ms
  mediasoup:ERROR:Channel [pid:23746] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- RESULTING TARGET SPATIAL LAYER: 0 +0ms
  mediasoup:ERROR:Channel [pid:23746] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- returning 1 +0ms

This should NOT happen:

  mediasoup:ERROR:Channel [pid:23746] RTC::SimulcastConsumer::RecalculateTargetLayers() | ----------- testing spatial layer 2 +0ms

[ibc]

It's not easy to fix. We must also take into account the scenario in which (as above layer 1 doesn't exist) and user is in layer 2 and changes preferred layer to 0, etc.