[Request] Midi support

[MaxGodTier]

Is there any way to send MIDI signals to Pedalboard? Most DAWs can forward MIDI signals to a VST plugin, then I can use Python as a virtual MIDI device to communicate with it programmatically, but haven't found any equivalent option for Pedalboard, no parameters seem to be for accepting MIDI signals (that I know of), implementing that would be the cherry on the cake! I'm absolutely amazed at the sheer amount of possibilities that are now at my fingertips thanks to your repo.

[psobot]

There isn't yet, but this is something that would be possible to add.

One minor complication: Pedalboard doesn't run in real-time on its own, so communicating with the plugin via MIDI would have to happen via some new API, rather than a virtual MIDI device. JUCE (the C++ library underneath Pedalboard) allows this kind of thing by effectively passing a list of MIDI messages in with every process call - something like:

my_plugin = pedalboard.load_plugin("...")

# Prepare a list of MIDI messages to send to the plugin as the audio is processed, of the format:
#     (time_offset_in_seconds, [midi_byte, midi_byte, ...])
# For example, this sends a single Note On message at a time index of 0 seconds:
midi_messages: List[Tuple[float, List[int]] = [(0.0, midi_note_on(C4, 127))]

output = my_plugin.process(audio, sample_rate=sample_rate, midi_messages=midi_messages)

This could work, but I'm considering if there are more ergonomic APIs that could be used rather than just overloading the process method with additional arguments.

[Muzz]

Just want to pipe up, and say that midi support would really be helpful here, and really open up the possibilities with this library.

Even just static midi rendering would be great.

[jorshi]

+1 !

[psobot]

Glad to see the interest here! Just to clarify: what kind of MIDI support are people looking for? It seems @MaxGodTier's request is around real-time MIDI, @Muzz is looking for static rendering of MIDI files, etc. This might be a number of different issues in one:

1. add support for virtual instrument plugins in addition to effects (i.e.: MIDI -> Audio instead of the current Audio -> Audio)
2. add support for MIDI file I/O and MIDI-to-Audio rendering (maybe via integration with Mido)
3. add support for sending MIDI messages to effect plugins during effect rendering (i.e.: Audio + MIDI -> Audio)

It'd be good for Pedalboard to support all of these eventually, but I'm curious to know which to prioritize.

[GabFitzgerald]

+1

really interested in virtual instrument plugins supporting MIDI -> Audio

[jorshi]

@psobot - thanks for following up on this!

Use case I would be most interested in would be static midi to audio rendering with VSTi plugins. I’m interested in ML research with synthesizers, so being able to programmatically adjust parameters and render sounds using simple midi files / objects would be ideal. There are a couple other python options out there that support this, but they aren’t pip installable. Having a pip installable package for VSTi rendering would be amazing!

[Muzz]

Yes to all of the above, though i feel that you should probably prioritize in terms of what functionality matches what is there already.

Offline vst instrument midi rendering is a little served niche so would be a great first stop.

[MaxGodTier]

It seems @MaxGodTier's request is around real-time MIDI

Actually I find it more convenient to render the results directly (faster) to a file than a real-time solution (slower), I just didn't know that was the easiest solution to implement! In my case scenario, the VST plugin is adjusting the pitch from an existing audio signal (. wav) to fit the notes being played from a midi track (.mid) like in this video I showed (wav file contains only static noise, mid file is playing music), I'm not sure if that's more like 2 or 3, however I don't require real time (it's actually better if it renders it to a file directly!)

[olilarkin]

+1 please support loading instruments :-)

[kenhepen]

+1 to all of three above.

I'm interested in sending midi messages to effect plugins in continuous mode. Example: I want to use pitch corrector and I want to send midi notes to it like: and as result I want pitch corrector to work continuously, like it works now when I just pedalboarding whole file with fixed settings. I mean not like cycle (https://github.com/spotify/pedalboard#does-pedalboard-support-changing-a-plugins-parameters-over-time) , because cycle with every iteration will reset parameters of pitch corrector and pitch of vocal will fluctuate.

So somehow whole pedalboard needs to "feel time", bars, frames, milliseconds.

[JourneyToSilius]

Same here, I would like to be able to read midi files that I generate and map them to VST synths as to create a full track

[valleyofblackpanther]

Is there any branch in this repo where this issue is being developed? @psobot

[psobot]

@valleyofblackpanther Somewhat - @robclouth has an open pull request that started to add MIDI message support, but that PR has been abandoned for some time and I don't know of any other efforts to add MIDI support at the moment. Pull requests are welcomed though!

[valleyofblackpanther]

@psobot do you think this python midi api could be used in the process of building the midi support. I think this api could be used to build the midi messages. But still I think it would be great if you could take a look at it.

[psobot]

@valleyofblackpanther If we were to build in MIDI support, it'd be great to use Python's duck typing avoid adding an explicit dependency on any specific MIDI library. MIDI messages are just tuples.