Add support for NoteOff-triggered modulators

[derselbst]

Related discussion

#1598

Is your feature request related to a problem?

The original use-case is explained in this comment. The problem is, that e.g. a flute sounds indefinitely while the sustain pedal is depressed after a noteOff has been received.

Describe the solution you'd like

A new modulator source Note-Off Velocity is to be introduced (with it's value yet to be defined).

As the name says, it's input value evaluates to the velocity of the noteOff event, ranging from [0;127]. The MIDI spec says that a noteOff velocity=64 should be used as "default" or "ignored" value.

The input value will then be mapped by the well-known (switch, linear, convex, concacve) mappings into [-1;1] range as usual.

Iff the noteOff event has not been received by the synth, the mapped input value of this source is considered zero, effectively causing the entire modulator to be disabled, as its effect evaluates to zero.

If a noteOn event with velocity=0 is sent, it shall be interpreted as noteOff velocity=64 as per MIDI spec.

A noteOff event with velocity=0 shall not receive a special treatment! Instead, the user defined mappings should provide the desired mapping.

Describe alternatives you've considered

Original and superseded proposal:

To address this problem, the user requires a modulator, that gets only activated after a noteOff event has been received. This would allow to manipulate the volEnvSustain or even adding further effects after receiving a noteOff event.

My preferred solution is to introduce a new modulator transformation. This new transform should evaluate to:

• either 1, when a noteOff event has been received, or
• 0 otherwise.

(Note that the behavior would be undefined for overlapping MIDI notes.)

Originally, the soundfont spec only supports two transforms: linear and absolute. A new transform - let's call it HasNoteOff - can be added, which is meant to be bit-wise ORed with the other two transforms.

The result of this binary transform (= 0 or 1) would then be multiplied with the modulator amount right before it's being added to its destination generator.

The sfModTransOper field in the Soundfont is two bytes in length, so we have plenty of bits to spare. I'm suggesting to define HasNoteOff = 0x8000 ie. setting the highest bit.

The spec also states that "Unknown or undefined values are ignored." which makes this solution backward compatible to existing implementations. In fact, fluidsynth actively disables modulators that use transforms it doesn't know about.

Additional context

Fluidsynth currently does not support or handle noteOff velocity. I.e. all API functions need to be duplicated (e.g. fluid_synth_noteoff2()) and shell commands need to receive the velocity as additional parameter.

[davy7125]

What I understand in #1598:

• the flute is played with no attenuation as long as the key is held (loop enabled and maximum volume)
• if the key is released while the sustain pedal is off, the sample enters in a 0.25 second release phase
• if the key is released while the sustain pedal is on, the sample enters in a 10-second release phase and if the sustain pedal is now off, the release time is shorten to 0.25 second.

If I am correct, with the proposed solution (either with a transform or with an extra modulator input), the sample volume may suddenly change when the key is released while the sustain pedal is on. We may indeed be beyond the decay phase when the sustain pedal is off with the scenario:

• play a key
• enable the sustain pedal
• wait more than 10 seconds (duration of the decay phase)
• release the key => here we will be in the sustain phase and the volume will suddenly drop with no slope
• finally release the sustain pedal

What would be needed is another playback mode: loop the sample until the key is off without taking into account the sustain pedal. The initial release duration would be 0.25 second and a modulator would change this value to 10 seconds when the pedal is on (we already have this feature).

[stgiga]

@spessasus @sylvia-leaf could this go into SFe?

[spessasus]

If I've read everything correctly, I don't think that the proposed solution (binary transformation) is a good idea and I think I have a better one:

A new modulator source: note-off velocity. It would evaluate to 0 if the voice is not in the release phase and to whatever noteOff velocity the message was transmitted if released.

MIDI note off messages do indeed specify the velocity of the note, and according to this website, if a MIDI doesn't want to specify a note-off velocity, it should use 64, so all APIs of any softsynths can add the parameter to noteOff and default it to 64.

This allows us to do cool things:

1. Detect if the note has been released: use an unipolar switch. 0 if on, 1 if off. And since 0 times the other source is still zero, the modulator is effectively disabled.
2. Negate the effect above (disable the mod if the voice is released): unipolar negative switch!
3. Allow the SF2 format to support note-off velocity. This could allow soundfont engineers to program in more accurate behavior to instruments, such as the rhodes piano:

Note-off velocity means the speed with which a pressed-down key is released back up. This information can be used for at least Rhodes electric piano sounds, so that the keys make a different release sound depending on if the keys are released softly or ”hard”. The difference in sound isn’t big, but there is a difference. An experienced player can notice the difference in sound, and loud key releases can be used for certain types of rhythmic “barking” comping.

So if we're extending the SF2 spec, we might as well kill two birds with one stone.

There is one problem with this approach: if a note-off with a velocity of 0 is reported, the modulator will evaluate to 0, which stands for "the voice is still on." I think that a good compromise would be limiting the minimum note-off velocity to 1, as I don't think it would make a lot of difference. But I'm not sure. What do you think, @derselbst?

[stgiga]

@spessasus I like the idea. Anything that can improve realism is a nice feature to have.

[davy7125]

@spessasus could you please read my comment above and give me your opinion?

Changing a parameter with values not being continuous would result in a noticeable break in the sound. A curve needs to apply and the current ADSR envelope doesn't provide it for this use case.

[derselbst]

Two notes from mine: Fluidsynth currently doesn't support noteOff velocity, which is why I tried to avoid this solution. I'm not voting against it - I actually do see the benefits - it would be just more effort to implement.

Davy has made a valid point with the sudden sound-break. Davy, could you elaborate a bit on the "playback mode" you've mentioned? Fluidsynth has adopted the "release" sample mode (#1397), but this one does not ignore sustain. I'm confused because you said "we already have this feature".

[mrbumpy409]

Supporting note-off velocity would be quite useful for emulating key release behavior of pianos and harpsichords. In both instruments, a damper is dropped onto the strings to silence them when the key is released, but if you release the key slowly, the damper takes longer to mute the strings. Quickly releasing the key can also result in more mechanical noise. Both of these behaviors could be easily emulated by having note-off velocity modulate release time and mechanical noise release sample volume.

Note-off velocity could also have fun applications in synth sounds, where you could have a faster key release cause a nice squelch as the filter release is quickened.

[spessasus]

Two notes from mine: Fluidsynth currently doesn't support noteOff velocity, which is why I tried to avoid this solution. I'm not voting against it - I actually do see the benefits - it would be just more effort to implement.

@derselbst I know, no SF2 synth I'm aware of supports note-off velocity. And this is a perfect opportunity to make them do so, while also making use of the flexible SF2 transform system, so it can act as a lock that prevents a modulator from working if a key is still on.

This is also why I mentioned the default note-off velocity: it can be used to expand a synth's API without breaking changes, like so:

// from
function noteOff(channel, midiNote)
// to
function noteOff(channel, midiNote, velocity = 64)

That way if an app does synth.noteOff(0, 60);, it will still work like normal, especially since by default there are no default modulators that respond to note-off velocity.

I could also quickly implement the modulator source i suggested into my soundfont editor and upload the compiled HTML here so we could test and refine the extension. Speaking of which, I also suggest that the modulator source for note-off velocity would be 1, as in the MIDI standard itself, note-off also comes before note-on in the numeric order. And that slot is free (0 - no controller, 2 note-on vel)

@spessasus could you please read my comment above and give me your opinion?

@davy7125 You raise a valid concern, as the modulator suddenly changing the sustain level from 144 (10s decay, to simulate a slow release with the hold pedal) to 0 (actual 0.25s release phase) would indeed cause a click. I'm not sure how the sample mode you've proposed would help this though, as it doesn't affect the volume itself. Stopping the sample's loop after the note is off (even if the sustain pedal is on) would just silence the sample, instead of the 10s decay we want. Unless I misunderstood your suggestion, that is.

[davy7125]

@derselbst @spessasus I proposed a solution for the specific original need: changing the release duration of the ADSR after a key is released, the sustain pedal being ON or OFF. I'll try to explain better:

• Another playback mode would be used, which is like the loop mode ON but not taking into account the state of the sustain pedal. As soon as the key is released, the voice enters in the release phase regardless CC64.
• The release phase duration is initially 0.25 second for quickly stopping the sound
• A modulator linked to the state of the sustain pedal would be used for increasing this release duration (this is where I said "we already have this feature"). This modulator, having as input CC64 ("hold pedal ON/OFF"):
  • changes the release duration from 0.25s to 10s when the sustain pedal is ON
  • changes back the release duration from 10s to 0.25s when the sustain pedal is OFF.

The result is that when the sustain pedal is ON and if the key is released, the voice enters in a release phase having a 10s duration and playing then with the sustain pedal will change the duration between 10s and 0.25s. Depending on the implementation of the effect of the ADSR on the volume, we may have this result:

Please note that this solution uses no modulator reactions to a note-off event => this is an attempt to solve the release duration change with a new sample playback mode: loop while ignoring CC-64. This first part could be in another ticket.

---

Now, returning to the main subject of this ticket and for giving you my thoughts about the note-off events:

• I would prefer the use of a note-off event as input of a modulator, even if the value of the event will be used in very specific configurations. The note-off velocity could still be used as mentioned by @mrbumpy409
• The transform should conceptually remains post operations. Only the absolute operation is implemented for now but we may find other useful operations like "sine" later.
• What is missing is: how fast should move a value driven by a modulator reacting to a note-off event? (this concern applies to all kinds of ON/OFF events). For avoiding sound breaks, the driven value - for example filter cut-off or attenuation - must have a smooth progression when the note-off event is caught. We are not in the case where a CC knob is turned: the knob sends contiguous values while the note-off event is a one-shot interruption.

In my view, we must find a way to handle abrupt value changes to keep the resulting sound musically acceptable. In a modulator, this could be an extra step after the arithmetic operation for specifying a duration, this duration being used for smoothly introducing the modulator modifications.

Or maybe specifying a duration here would be too much of a hassle, from both the user and the developer points of view: the user may have no idea how slow the change should be and the soundfont specifications prevent us from adding an accurate duration in a modulator description. So maybe the post operation could simply be a combination of:

• absolute value ON / OFF (first flag)
• smooth transition ON / OFF (second flag).

The duration of the transition is yet to be defined as a standard and maybe 0.1 second would do the job.

[davy7125]

I may have a better solution to the duration of a modulator change. Each modulator has two inputs and for each input it is possible to specify the type / polarity / direction:

The type can be one of the following:

• 0 - linear
• 1 - concave
• 2 - convex
• 3 - switch

2 out of 6 bits are used and we could:

• take a third bit as a flag for requiring a smooth duration (0.1 second for instance)
• or take the 4 other bits for specifying a duration with for example this rule:
  • 0 => instant change
  • 1 => 0.01s transition
  • 2 => 0.04s transition
  • 4 => 0.16s transition
  • 8 => 0.64s transition
  • 15 => transition around 2.5s

This will be the duration to move from 0 (unipolar) or 64 (bipolar) to the received value. This duration would be added to the left part of a generator definition, next to every curve:

These thoughts can be a third ticket for better defining the work to be done:

1. playback mode "loop and ignore sustain pedal"
2. note-off event as modulator input
3. transition duration associated to a modulator input

[stgiga]

We should see what Fluidsynth wants.

[spessasus]

I think that adding volume smoothing is a bit too complex and unnecessary, especially since some SF2 synths (such as BASSMIDI or spessasynth, not sure about fluidsynth) already smooth out volume changes, and adding another layer of smoothing would be a bit messy. Not to mention that it kind of feels like a band-aid solution, doesn't it?

Also redefining the curve type based on the source is also a bit too extreme, as for example transformType no longer is an accurate description of the field, as it also controls transition time.

It would make modulators even more confusing for sf2 softsynth developers, which is already a problem right now.

Instead, I have another solution, involving only two additions: the note-off velocity and a release generator.

---

The problem

The main problem is that you're trying to use the decay phase as the release phase, which is not really suitable here. So what if we control when the release phase happens instead?

Controlling the release phase

We could assign an unused generator (my suggestion being 59 - unused5, as synthfont uses the lower ones for its own proprietary extensions) to something like isInRelease or released. It has a value of either 0 or 1. If 0, the voice is not released, and if 1, it is released. It can only be released once.

Default behavior

This generator fits nicely with our note-off source, as we can program in the default behavior, like so:

• s1: note-off velocity unipolar switch positive
• s2: sustain pedal unipolar switch negative
• amount: 1
• destination: 59 (isInRelease generator)

This perfectly replicates the regular MIDI behavior: OFFing the voice if it has a note-off velocity (released) and the sustain pedal is off (hence the negative transform).

This modulator would be a default modulator. But it can be disabled, which we will make use off below:

The specific use-case

We can program in the desired behavior for our flute:

1. Disable the default release modulator (copy everything but set the amount to 0). This disables the sustain pedal.
2. Add a modulator that ignores the sustain pedal:
   • s1: note-off velocity unipolar switch positive
   • s2: no controller = 1
   • amount: 1
   • destination: 59 (isInReleaseGenerator)
3. Program in the different release times:
   • releaseVolEnv generator is set to 0.25s
   • s1: sustain pedal unipolar switch positive
   • s2: no controller = 1
   • amount: (9.75s in timecents)
   • destination: 38 (releaseVolEnv)

This way:

• the voice's state is set to release, regardless of the sustain pedal (which is what we want)
• release is 0.25 if the pedal is off, 10s if the pedal is on (releaseVolEnv is evaluated once, on note release. So we'd have to keep in mind that first we evaluate modulators, then check if they released a voice. If they did, then compute the releaseVolEnv)

---

Advantages of this approach:

1. Hold pedal is no longer hardcoded. The sf2 musicians can get creative!
2. Only 2 additions: source and destination. No new sample modes.
3. There are no clicks (we're just changing the volEnvRelease time)

@davy7125, @derselbst what do you think?

[stgiga]

I think that all of these ideas have some form of merit to them. Ideally the approach with the least downsides and the most upsides would be wise to choose, even if it involves a hybrid approach. For instance there's a LOT you could do if you used empty mode bits to do stuff like bidirectional loops hypothetically, and there are other parts of the proposals in this thread that each approach mentioned has that can improve realism or synth dynamicness. I think that using the all 4 bits for timed intervals may be too limiting. The stuff about note-off velocity would help harpsichords and pianos. Also we don't want clicks.

[RobsonFBP]

What if the DECAY2 phase was created in the envelope as described in this commen?

[derselbst]

Thanks to all for this lively discussion so far! I do perceive that noteOff triggered modulators have "hit the spot". You guys have also convinced me that going with noteOff velocity as input source being more beneficial as more use-cases could evolve from it. I will shortly adjust my initial post to reflect this.

To solve the original use-case from @RobsonFBP, I'll need to think a bit more about Davy's proposal vs. Spessasus proposal, with the latter seeming slightly more easy to implement.

:warning: Both proposals have in common that you'd manipulate the release duration while the voice is already playing, which I'm afraid fluidsynth is currently not capable of.

:warning: Also note that both proposals would require CC64 to be allowed to modulate, though the spec says it's not allowed to modulate (that IMO is a negligible detail)

✔️ From my understanding, both proposals could also be easily applied to Sostenuto (with Spessasus proposal providing a bit more flexible user-defined handling of Sustain vs. Sostenuto, while Davy's new sample mode would be implementation-bound to either Sustain, Sostenuto or both).

---

Regarding the question of smoothing out things, I would actually not take any action but rather wait and see if this becomes an issue. I am confident though it won't happen. Fluidsynth already smoothes out volume changes as well as FilterFC and FilterQ changes.

---

This is also why I mentioned the default note-off velocity: it can be used to expand a synth's API without breaking changes, like so:

@spessasus Adding a new parameter is both, API and ABI breaking, even if it has a default value ;)

[spessasus]

Both proposals have in common that you'd manipulate the release duration while the voice is already playing, which I'm afraid fluidsynth is currently not capable of.

Looks like you didn't understand my proposal because it's the exact opposite: the idea is for the synth to evaluate release time once, when the voice's state is set to released. Allowing it to be changed while the release is happening wouldn't make a lot of sense and it would click a lot. You can't "catch" a released note with a hold pedal.

So, an example where a hold pedal is on:

1. Note off is received.
2. Modulators are evaluated.
3. The synth checks voice's modulated isInRelease generator.
4. It's now set to 1, so the voice is set to release.
5. The modulated volEnvRelease is equal to 10 seconds, since the hold pedal modulator was also evaluated.
6. The release phase is set to 10 seconds and it begins.
7. Even if the hold pedal is released while the voice is playing, the release is still 10 seconds as desired.

There's no real-time change.

@spessasus Adding a new parameter is both, API and ABI breaking, even if it has a default value ;)

Well, if I did noteOff(channel, midiNote, velocity = 64) in JS, that's not a breaking API change (as old 2 parameter calls work just fine), so I guess C is different in that regard :-)

[stgiga]

Personally I feel like Creative restricting modulator CCs was a mistake given you could do rather interesting things with them. I sort of get wanting to prevent modulators running Bank Select, though even doing THAT has uses. Also in theory a real SB card would just ignore denylisted CCs the same way it reacts to CC94 Insertion EFX. It's as minor of an interpretation as doing Bank Select LSB via wBank's other byte, putting the sample chunks at the end to allow ~8GiB banks without RIFF64, making 8bit banks using orphan sm24 chunks, doing 32bit via a new sm32 chunk (SF2 2.01 and 2.0 had subchunk restrictions that SF2 2.04 lifted to add 24bit support, so in the sample area of SF2 structures, like the INFO chunks you can do whatever you want.) Basically all of these involve taking what the standard says literally enough to not do ONLY what is specifically said in the standard to specifically reject as malformed, and only MUST NOT is a no.

Basically me and the rest of SFe have dug into the standard and using every possible opening accidentally given, added to the format in ways Creative never thought of.

The SF2 format has link types other than L/R, meaning you could with panning do surround presets. Also we're planning on using Genre, Library, and Morphology tags that went unused, some additions from Silicon SoundFonts (Section 11 of SF2), the unused byte of wPreset, and other stuff that Creative was accidentally forward-thinking about by making a spec in which one can have a LOT of fun if clever, something like my own formats to be honest.

[RobsonFBP]

@davy7125

To achieve this functionality in @davy7125's graph, we need to trigger the release phase whenever the hold pedal state changes, because with a time change, the countdown always starts from the beginning of the release. Therefore, if the pedal is pressed (CC-64 = 127), the release time jumps from 0.25 s to 4 s, and if the pedal is released (CC-64 = 0), 2 s after releasing the key, the sound suddenly stops, because the release trigger was already 2 s ago and the release time is now 0.25 s.

Therefore, in addition to changing volEnvRelease (which the modulator will do) whenever CC-64 changes, we need to trigger the release function again to initiate a new release when the CC-64 status changes from the current point.

@davy7125

this is an attempt to solve the release duration change with a new sample playback mode: loop while ignoring CC-64.

I don't think you even need to create a new playback mode. We just check if a modulator like this exists for the voice:

Source 1: noteOff Velocity Source 2: CC-64 Dest: volEnvRelease

If it exists, we ignore the sustain, passing the volume envelope to the release phase.

@spessasus

Well, if I did noteOff(channel, midiNote, velocity = 64) in JS, that's not a breaking API change (as old 2 parameter calls work just fine), so I guess C is different in that regard :-)

I think this can be worked around by having the old function call the new function, maintaining compatibility.

// new function noteOff_vel(channel, midiNote, velocity)

// old function call a new function function noteOff(channel, midiNote) { noteOff_vel(channel, midiNote, 64); }