Export to WAV, MP3

[daveyarwood]

Moved from https://github.com/alda-lang/alda/issues/62.

See the original issue for some ideas on how to do this.

[daveyarwood]

The AudioSynthesizer class looks promising.

There are various Midi2WavRenderer implementations out there that use it to render a Sequence to an audio stream.

How we would integrate this with other (non-MIDI) types of Alda instruments in the future is an open question, but in theory, it seems like it ought to be possible to render multiple audio streams (MIDI and non-MIDI) and combine them into a single audio file.

Another open question is whether we will be limited to the Sun Java runtime if we go this route. If so, I think it might be worth it.

[daveyarwood]

If we are limited to the Sun Java runtime, maybe one way we could make things easier for the end user would be to bundle the Java runtime into the executable.

[jgkamat]

I really think it's rediculous that alda dosen't have any form of export, and a bunch of my friends needed alda to have an export feature for a hackathon project, so I spent a bunch of time messing with this. Right now it's terrible quality (which is why it's not in a PR) and not even close to completion, but it does work, somewhat.

I'll try to encode everything that I learned here, since resources on java midi export are incredibly rare (?!?!?)

Very basic googling will quickly center to this page, which describes the method java uses for midi recording and export. It's a little tough to understand, but it's better than the API (although still not great). Any "MidiDevice" has support for both 'receiving' and transmitting' audio, if set up correctly. Two relevant subclasses of MidiDevice are Sequencer, which is able to write and read from a sequence, the "official" method of storing midi "scores" in java. The second is the Synthesizer, which is capable of producing sounds when written to or when commanded manually via function calls.

The Midi Wave exports mentioned before all work with sequences (they read midi files from disk into a Sequence) so I wanted to re-direct the Synthesizer to 'transmit' into a Sequencer 'receiver'. However, when you try to do this, you get a MidiUnavailableException, which lead me to a dead end (no one seems to have overcome this, there are several SO posts with no answers with the same problem as me). While I haven't looked at the source, the 'getMaxTransmitters' function returns 0 on a Synthesizer, leading me to believe this is route is impossible (if someone knows how to do this, please let me know).

Because of that, I decided to go the second route, which is building Midi Sequences from alda scores manually, then writing those scores to disk. I'm actually a little bit confused at the design for this system. It seems to be scheduling every note start and note end via a callback (via jsyn), and using those callbacks to call functions on the midi Synthesizer. Isn't this fragile (and can lead to timing problems?) From what I can tell, java highly recommends building a sequence if timing is important.

This web page was incredibly helpful during this process, I probably couldn't have done anything without it.

Essentially all I'm doing is looping through all the events in a score, and sending corresponding notestart and noteend events (instead of registering callbacks) into a sequencer. Finally, I tell the sequencer to record what I'm transmitting, and it builds a midi sequence for me! Writing it out to a file then is fairly easy, and converting it to a wav is just one more step which should be easy (I didn't do it for now, since I was able to just launch another java process to do that for me).

One of the problems with this method is that (because I didn't understand how instruments work) all the events are played with the piano. However, I think this should be pretty easy to add with a bit more playing around.

Also because I was super crunched on time, it encodes midi's whenever it plays, to a hardcoded file, but that's not really the point of this :P.

I don't know if there's another way to do this 'cleanly'. I really despise listening to audio out (since recording much be done in realtime, and it feels really hacky), so I think that midi export should be a feature in alda even if jsyn features are added (for pure midi tracks). I couldn't find any way to write jsyn out to a file, so I don't know how much luck we'll have with that.

Would you like this to be developed further (because it does subvert the core internals of the alda player), or do you think there's a better way to do it? If you think it's good approach, I'll clean it up (eventually......) and make a PR with proper integration in the client/server as well (but this will probably take a while).

Just to prove it works, here's a midi (piano only, as mentioned before) of my favorite alda score ever!

potforanything.zip

[daveyarwood]

@jgkamat Wow! Thanks a lot for digging into how all this stuff works. I think this is a great first step for this feature.

I took a brief look at the code on your branch, and in general I think it looks like the right direction to go in. It's very encouraging that you were able to get something roughly working already!

Some quick comments:

• Using a Sequencer instead of a Synthesizer is definitely what we want to do, at least for the purposes of export.

• Your confusion about us using a Synthesizer to produce sound is totally reasonable. I'm not 100% sure that what we're doing now is the best approach. It might be better if we created a Sequencer and loaded all the events into it, something like what you're doing on your branch, and then we would be free to either output to a MIDI file or play the sequence via the Java MIDI Synthesizer.

• We are using a library called JSyn to do high-precision audio scheduling, which is why using a Synthesizer works for us without running into note timing issues. I ended up doing it this way because JSyn gives you a highly precise, general purpose event scheduling system. The general purpose part is key, because I eventually want to add non-MIDI instrument types (see: alda-lang/alda#341, alda-lang/alda#338).

• I think it would be worthwhile for us to explore / experiment with using a Sequencer to do all of the scheduling. I'm curious if there is some kind of custom MIDI message we could send that we could use to schedule non-MIDI note on/off events when performing a score that uses non-MIDI instruments. We don't have anything like that yet, but we do have scheduled function events, so perhaps we could use those to test. If we can use a Sequencer to (for example) print "hello" at a precise moment in a score, then I think we'll have what we need.

• I'm also curious if the Sequencer's timing is as precise as JSyn's scheduling system. I imagine it is, but if there are any noticeable timing issues when playing a sequence, that would be a deal-breaker, and would indicate that we should stick with the Synthesizer / JSyn combination for better timing. I suspect that it's pretty accurate, though.

• I think that exporting a score should be a separate action from playing it. I'm imagining running a command something like alda export -f myscore.alda -o midi > myscore.mid, and the export would happen as quickly as possible without also playing the score. Is it really the case that to record a sequence, you have to wait the length of the sequence? Just from looking at your code, it doesn't look like you have to do any sleeping or anything like that -- if I'm understanding this right, it just looks like you're spinning through each event in the score in a doseq and adding each one to the sequence, then writing the sequence out to a file at the end. I would expect that to be very fast.

• We can worry about incorporating non-MIDI instruments later. For the purposes of exporting to a MIDI file, we can just ignore them or throw an error or something. When it comes time to implement the "export to wav" piece, I imagine what we'll do is export the non-MIDI instruments separately via a different process (hopefully one that doesn't involve waiting the length of the score), and then merge the MIDI and non-MIDI audio streams together somehow.

---

So, at this point, I can think of a few possible next steps:

• An experimental PR that replaces the JSyn scheduling system with a Sequencer, and then uses a Synthesizer to play the resulting sequence. We can defer any non-MIDI events (e.g. scheduled function events) for now.

• Building on that, an exploration of how we might implement custom scheduled events using the Sequencer for timing. This would basically just be for the purposes of scheduling things that happen during playback (e.g. printing messages); no need to worry about exporting these custom events in any way.

• Building on the previous PRs, another small PR that implements "export to MIDI" as just writing the sequence to a file. (Or maybe it prints the bytes to STDOUT so they can be redirected to a file, in true Unix style.)

[jgkamat]

Thanks! :smile:

We are using a library called JSyn to do high-precision audio scheduling, which is why using a Synthesizer works for us without running into note timing issues. I ended up doing it this way because JSyn gives you a highly precise, general purpose event scheduling system.

Ah that makes a lot more sense now! I think it's fine to leave the existing JSyn scheduling system (since it's more extensible) unless timing issues are found (which they havent). It's extremely trivial to get a Sequence played back via java midi, so maybe I'll add a switch to choose which one to play back from (once this is done).

I think it would be worthwhile for us to explore / experiment with using a Sequencer to do all of the scheduling. I'm curious if there is some kind of custom MIDI message we could send that we could use to schedule non-MIDI note on/off events when performing a score that uses non-MIDI instruments.

I really doubt this is possible, which is why I think that we should probably stay with JSyn as a primary. Making a sequence is essentially just writing a MIDI file out by hand, so encoding custom things would probably not be possible (unless they are in the midi file format). You can take a look at ShortMessage to see the 'official' things that are supported. I think writing raw MIDI control codes works as well.

Is it really the case that to record a sequence, you have to wait the length of the sequence?

Nope, this isn't tied to playback at all, and saving to midi is incredibly fast! The song still plays afterwards (because it's a hack), but the MIDI is fully exported before the song even begins playing. For longer songs I do notice a delay (I'm not sure what the most expensive part is), but it's nowhere near the length of the song.

I think that exporting a score should be a separate action from playing it. I'm imagining running a command something like alda export -f myscore.alda -o midi > myscore.mid

Nitpick: I really don't think we should output to stdout by default (since the target is non-programmers, and exporting is something everyone needs to do). Running without the redirect would cause binary to be spit out, which is annoying. I would feel a lot more comfortable with a alda export --input myscore.alda --output myscore.mid --format midi (the flags don't matter so much), and adding a flag to output to stdout. That's a pretty minor detail though, and we can figure it out (and maybe have a poll) later.

On Next Steps:

I think replacing the JSyn Scheduling might be too big of a change for something this untested (and obscure) and it's not really needed at first. I'll (try to) implement a Sequencer+Synthesizer playback mechanism, and make it possible to switch between them with a constant flag. That way we can test out the Midi export easily, and still have the existing method (in case it's useful for non-midi audio). Later on, we can always decide to remove one.

I don't think custom events is possible with Sequencer (but I definitely could be wrong), which is one of the reasons I don't want to touch the existing playback code for now.

So my priority will be figuring out how to encode different instruments into midi (since everything else hinges on that), and once that's done, the rest should be fairly straightforward! :D

[daveyarwood]

That all sounds totally reasonable to me!

Thanks again for looking into this. Feel free to ping me here or on Slack if you want to discuss things further!

[nblumoe]

There is mp3 and wav export in https://github.com/oakes/edna, which is based on alda: https://github.com/oakes/edna/blob/ceb9ebd9b2d05169e008793a57b5bca64603cbb7/src/edna/core.clj#L201

[dirkk0]

A different approach might be the conversion to midi, and a subsequent conversion to wav/mp3.

I just tested this and it works:

alda export -f test.alda -o test.mid

# brew install fluidsynth sox
fluidsynth sf.sf2 test.mid -F raw_audio # -g 1.9
sox -t raw -r 44100 -e signed -b 16 -c 2 raw_audio out_midi.wav

[daveyarwood]

That's great! I still think it would be useful if Alda had a "batteries included" way of exporting a score to WAV or MP3 built into Alda itself, but it's awesome that there is a workflow that folks can use in the meantime. I'll probably use this, myself.

[fwindpeak]

A different approach might be the conversion to midi, and a subsequent conversion to wav/mp3.

I just tested this and it works:

alda export -f test.alda -o test.mid

# brew install fluidsynth sox
fluidsynth sf.sf2 test.mid -F raw_audio # -g 1.9
sox -t raw -r 44100 -e signed -b 16 -c 2 raw_audio out_midi.wav

Thank you! I found that the file output with fluidsynth plus the extension is a playable file.

Tested on a mac.

fluidsynth ~/.gervill/soundbank-emg.sf2  helloworld.mid -F helloword.wav

Supported file types:

$ fluidsynth -T help
-T options (audio file type):
   'aiff','au','auto','avr','caf','flac','htk','iff','mat','mpc','oga','paf','pvf','raw','rf64','sd2','sds','sf','voc','w64','wav','wve','xi'

auto: Determine type from file name extension, defaults to "wav"