Add support for playback/synthesizer

[kankaristo]

Thanks for the plugin! This will be very handy for my music theory notes.

I'm not sure if this is possible in Obsidian (or how much work it would be), but the abcjs library homepage shows an example of synthesizer support: https://paulrosen.github.io/abcjs/

It would be amazing to have this in Obsidian as well (an integrated player under the notation).

[TilBlechschmidt]

According to my knowledge that should not be a problem. Currently, the plugin only replaces the code block HTML element with the statically rendered svg element. However, it shouldn't not be complicated to append a media tag below it.

I'm unsure how Obsidian handles the AudioContext API and the topic of sound fonts comes up as AbcJS loads them via the network by default. It would make sense to bundle a default one so that it works offline. An option to change the sound font would be a nice extra.

Leaving this here for later reference: https://paulrosen.github.io/abcjs/audio/synthesized-sound.html

One question @kankaristo, would you want a player below every sheet or only in some places (e.g. by changing the language or adding a prefix like obsidian:playable to the top of the code block)?

[TilBlechschmidt]

One thing that comes to mind: Obsidian does not render everything on a note all the time so the ignorant solution of just appending an HTML audio player tag as per abcjs docs would cause some issues when you scroll it out of view (as in it would stop playing — might also be considered a feature though).

[kankaristo]

One question @kankaristo, would you want a player below every sheet or only in some places (e.g. by changing the language or adding a prefix like obsidian:playable to the top of the code block)?

That depends on how large the player would be. A small player could probably always be there. But if you only have short notation (e.g. for a single scale), and the player takes up as much space as the notation, it might look a bit funny.

So, a prefix to enable/disable the player per score might be the best option.

Even better (maybe not for the first version) would be 3 options:

• no player
• simple player
  • single play/pause (or play/stop) button only, in the corner of the score
  • would be perfect for short notation, where the entire thing is up to 5 seconds long)
• full player
  • play/pause, seek bar, jump to beginning, etc.
  • similar to the example at abcjs

I might take a look at implementing this, but I'm not sure when I'll find the time.

---

One thing that comes to mind: Obsidian does not render everything on a note all the time so the ignorant solution of just appending an HTML audio player tag as per abcjs docs would cause some issues when you scroll it out of view (as in it would stop playing — might also be considered a feature though).

That might be a nice feature, I probably wouldn't mind. Unless you have a really long score, and scrolling inside the score makes the playback stop... As a workaround, you could just split into 2 panes, play the audio in one, and scroll the document in another.

If the player is inside the same parent element as the SVG, Obsidian probably shouldn't remove it while the SVG (even a big one) is on screen? But I'm not familiar with Obsidian's plugin API, so I'm not sure if this is possible, or if the player has to be put into it's own line/parent element.

This could also be considered a bug in Obsidian (Obsidian shouldn't remove a playing audio or video element from the DOM). So, if it becomes an issue, it might be fixed in Obsidian (no guarantee though).

[tdresser]

Would a fixed position "currently playing" indicator be possible with the obsidian plugin API? I think that would ideal - if you start playing something, and then scroll away, you'd still like to be able to stop what's currently playing.

I'm not sure how obsidian determines what to drop from the DOM, but perhaps this would both be good for UX and enable the audio to continue playing?

[TheMathGuyd]

Hi! I know that this project might not be very active (since its current features are so stable), but synthesized playback (of any kind) from anyone who knows how to implement it would be very appreciated.

[TilBlechschmidt]

Hey @TheMathGuyd, since I am not using Obsidian (or this plugin for that matter) actively it is not very high on my priority list. For this reason nice-to-have features like this are not making much progress. However, if anyone wants to contribute I would gladly invest the time to review the changes and provide feedback! 🙂

[mbvs]

I gave this a shot and I've got a functional POC running (https://github.com/mbvs/obsidian-plugin-abcjs/tree/feature/synth), but I am a little unsure about the details, i.e. what features I should spend time on.

my findings so far:

• soundfonts need to be in the MIDI.js format, only 5 exist so far (https://github.com/paulrosen/midi-js-soundfonts) but it would be possible to convert some sf2 to this format (something from https://sites.google.com/site/soundfonts4u/) Right now the scores are analysed and the necessary samples loaded and cached. The fonts include the complete GM instruments, so they are big and nevertheless bad sounding. Bundling with the plugin would make the install of the plugin unbearable slow. I also have not tried yet to load the samples from the filesystem, not sure if it is possible at all.

• the included audio control widget looks "okayish", would be nicer to have something looking similiar to the html audio control, but this would be a lot of work.

• the cursor currently jumps from note to note - I would prefer something continously animated, but this also seems to be a non trivial work.

I intend to use this plugin to learn music-theory. So I need only short snippets (scales, chords, cadences ...) Also no need for full GM soundfont, I would prefer a single instrument (piano) but better sounding.

What do you think the scope of this plugin should be? What uses do you imagine?

[TilBlechschmidt]

First off all, thanks for taking this on @mbvs! 🙏

Regarding the findings:

1. If possible it probably makes sense to bundle a default, small soundfont and then allow the user to provide a path to a custom soundfont on disk — either per codeblock in the settings object or through a settings UI. I definitely would not go with the bundling approach. If loading from disk is not possible, maybe providing URLs might work (maybe even file:// URLs?)
2. I'd say for a initial feature release having the default audio controls is fine
3. Same as above, while I agree in that a nicely animated cursor would be neat, it is not "necessary"

Regarding scope I honestly have no hard constraints though I would refrain from adding too many large, use-case specific features that depart from what abcjs itself provides — mostly to keep it maintainable / approachable. Though if someone makes a good case for a feature I'm not going to stop them 😉

[mbvs]

Okay Til, will keep working on it as time permits. Although Paul Rosen's code isn´t the most obvious, I am having fun (and I want to use the plugin :wink:). When I've got something presentable I will let you know.

[SimplGy]

Just discovered ABC notation, very excited. Love the rendering with tabulatures, would love music synthesis to help my practice.

I have the JavaScript (10+ yrs) and Obsidian plugin (2 so far) background to be able to help land this feature.

May I?

@mbvs you ok with me carrying the torch? @TilBlechschmidt around to review a PR this weekend?

[alensiljak]

I have the JavaScript (10+ yrs) and Obsidian plugin (2 so far) background to be able to help land this feature.

Try it out. It may be as simple as turning on a feature flag in abcjs. Or it may prove to be impossible due to incompatibility of the shell, as is currently the case with the VS Code extension.

[mbvs]

go ahead - I got it working, was pretty easy, but got lost in the code for a floating cursor. Worked so far, but no possibility to deliver soundfonts with the plugin. Finally gave up Obsidian and wrote some JavaScript to render me Anki-cards which prooved more usefull for my usecase.

[mbvs]

btw, there is another abc-notation software by Jeff Moine which imho renders nicer. Also interesting is abcWeb by Wim Vree

[alensiljak]

btw, there is another abc-notation software by Jeff Moine which imho renders nicer. Also interesting is abcWeb by Wim Vree

They all use different technologies and have their specific pros/cons. For my sheets, none of the ABC software renders really well (compared to Muse Score and Lillypond) but abcjs comes close.

[SimplGy]

Did some playing around today and wrote up learnings in a blog post and associated JSFiddle. Surprisingly tricky to "just play music" in a browser. https://www.simple.gy/blog/generating-music-from-text-notation/

Still possible the Obsidian environment will make doing this impossible, but I have a fallback idea (a play link that opens a web based player), so will try both this weekend.

[alensiljak]

Yes, have a look, perhaps, at the vscode plugin, also. Exporting an HTML page and opening it in a browser is how I've worked around these limitations for printing. The concepts are similar, just a different environment/shell.

[SimplGy]

Looks like it's no problem for Obsidian to create and use an AudioContext, so standard abcjs options and synth stuff work great.

I tried several variations, would be interested to know what the maintainers would like to do:

1. No playback controls (click on the entire song area to start/stop)
2. custom/simple "play" button in the bottom right
3. Standard abcjs controls in the bottom right, but only when the song is hovered (shown below)

Here's what it looks like:

Features:

• loop/play/speed (from standard abcjs SynthController UI)
• highlight current note
• highlight song being played

Questions:

1. Does "enable song playback" need to be an option, or is it always available for users?
2. Which method (1-3) should I use to visualize/display the playback controls?

[alensiljak]

That's great to hear!

I personally like a simple play/pause button option. The remaining settings, including enabling the feature(!), might go into the settings. Cluttering the (usually) little space available in a note, especially for those who don't use the feature, should be minimized. Particularly important on mobile devices.

There's a wide range of use-cases we should be aware of. The plugin might be used to render a simple one-bar melody/rhythm samples, to having whole songs stored with multiple parts, instruments, and voices. As there are existing full-scale editors available, I would rather not turn the plugin into one. We should probably try to keep close to the goals of Obsidian, which is a note-taking, personal knowledge base, type of tool. A full editor functionality might be provided by specialized tools and melodies copy/pasted into Obsidian. Even the playback seems a bit of a stretch in case it turns out that it needs to download a huge sound font. In that regard, it might be a good idea to download and cache the font asynchronously once the feature is enabled, for example (if possible).

As far as the sound fonts go, I believe there is some default (software) MIDI functionality provided by the OS? It would be ideal not to have to download any extra assets. Users might also want to use different sound fonts. For example, I have some huge sound fonts that have various drumsets sampled. Doubt that these would be useful for most others. And vice versa - I really don't care about the quality of the piano sound in most cases and would rather not download a couple-of-hundred megabyte sound font automatically with this extension. Particularly if I only need to jot a few rhythms, but this might be important to someone who has classical piano melodies in their notes.

These are just my views. Hope others would chip-in with additional ideas and use-cases. That said, great exploratory work! Try to build something that will inspire others and make ABC more accessible and popular.

@paulrosen might also have some ideas/suggestions.

[tdresser]

Re "Standard abcjs controls in the bottom right, but only when the song is hovered (shown below)" Ideally we'd have something which works on mobile, so we should likely steer away from relying on hover.

[SimplGy]

Taking into consideration the feedback, now implementing the simplest possible playback, which is just "push to play, push to pause". Will work on all platforms that support sound. PR on the way.

I think we can probably wait to add options (like defaultQpm and chordsOff) later, in plugin settings.

I also looked at local sound font files, but did not add them yet. I think both are true:

1. we can always add later
2. they are large

Here's working sound playback, with correct start/stop and cleanup when you close tabs or go into edit mode:

(demo includes two songs in the same file just to test that case)

[SimplGy]

ptal, I think this is ready to go and am eager to use it on other machines.

[alensiljak]

This works great, thank you @SimplGy ! The feature is now available in v1.4.0.

[andrewortwein]

Just did a quick test and it's working well. Guess I should dust off some old songs and put them into Obsidian now. Thank you for doing this!