SuperDirt

SuperDirt copied to clipboard

It would be amazing if Tidal could use the Ambisonic Toolkit to control a sounds azimuth and altitude and encode/decode B-format for Ambisonics performance.

With some knowledge of ambisonics you could easily write a panner variant that uses it. The panning function is live hackable …

I’ll be spending some time in Dec working with 7th order HOA and WFS systems at a research center. We are going to work with live coding (tidal and hydra) and ambisonics. My live coding work now routes audio from SC into virtual channels in Live where I’m using the Envelop ambisonic tools. It is a very cumbersome and inefficient system - and lacks the dynamics of a pure live coding paradigm where I could address azimuth and altitude directly. Nothing against Envelop as they are great tools but I must run three extra applications to use them.

Just curious, if there might be a possibility to realize a Tidal function that could directly access SC’s ambisonic add on library by dec?

With some knowledge of ambisonics you could easily write a panner variant that uses it. The panning function is live hackable …

Curious what you mean. I personally don’t have any experience with the math behind ambisonics and building a decent encoder/decoder is definitely not in my skill set.

Hi @chronopolis5k, didn't you link to an already existing encoder and decoder above?

Hi @chronopolis5k, didn't you link to an already existing encoder and decoder above?

Yes, sorry - my mistake. I was extrapolating beyond what @telephon suggested, which is, from what I understand, modifying the panning function? That could address azimuth I assume? But there are other params such as Altitude and Radius.

Admittedly, I haven't wrapped my head around how the pieces would fit together between tidal, sc and the ambisonics toolkit. I have some time to look into it this week and will come back when I have some more clarity there.

In a way tidal doesn't work in the sound domain at all. It is just for patterning the parameters that are sent to superdirt. Adding a parameter to tidal is just a matter of running e.g. altitude = pF "altitude" So I think it's mostly plumbing in superdirt to use ATK for panning and route the extra parameters to it. I believe the existing panning stuff already receives extra parameters for multichannel panning such as splay

I was extrapolating beyond what @telephon suggested, which is, from what I understand, modifying the panning function? That could address azimuth I assume? But there are other params such as Altitude and Radius.

You can do it yourself, it's all in the open.

For a start, you could just write one SynthDef that uses the ambisonic toolkit and run it from tidal. The total number of channels for SuperDirt need to be correct for that.

Once this works, you can roll your own panner:

DirtPan.defaultPanningFunction = { |signal, numChannels, pan, mul|
   var altitude = \altitude.ir(0); // get extra arguments like this for event-based stuff (set param once)
   var altitude = \altitude.kr(0); // … or for global effects (change param while running)
   etc..

}

I see. Thanks for thanks for all the info. That helps clarify things a bit as I haven’t done much research into how things are configured. Whatever I come up with, I’ll share it back here. Cheers.

This is what I have so far. Haven't tested it yet. For some reason my params.hs is a binary file? It's actually a .hi so I cannot edit it. Any suggestions?

(
~dirt.addModule('amb-panner',
	{ |dirtEvent|
		dirtEvent.sendSynth("ambpanner" ++ ~numChannels,
			[
				azim: ~azim,
				alt: ~alt,
				radius: ~radius,
				out: ~out
		])
}, { ~azim.notNil or: { ~alt.notNil }});
);

(
{
var numChannels =  ~dirt.numChannels;

SynthDef("amb-panner" ++ numChannels, { |out, azim = 0pi, alt = 0.2pi, radius = 1 |
	var signal = In.ar(out, numChannels);
		#w, x, y, z = BFEncode1.ar(signal, azim, alt, radius);
	//decode for 2 channels, binaural
	BFDecode1.ar(w, x, y, z, [-0.25pi, 0.25pi], 0);
	ReplaceOut.ar(out, signal)
	}, [\ir, \ir, \ir, \ir]).add;
}
);

Hi @micah-frank-studio, did you get further with this?

During lockdown I'm interested in being able to do streamed binaural performances, to have more presence for people listening on headphones.

I'm currently trying this:

	~dirt = SuperDirt(2, s); // two output channels, increase if you want to pan across more channels

	DirtPan.defaultPanningFunction = #{ | signals, numChannels, pan, mul |
		var channels, inNumChannels;
		#w, x, y, z = BFEncode1.ar(signals, \azim.ir(0), \alt.ir(0), \radius.tr(0));
		BFDecode1.ar(w, x, y, z, [-0.25pi, 0.25pi], 0);
	};

	~dirt.start(57120, [0, 0, 0, 0, 0, 0, 0, 0, 0]);

with this:

d1 $ sound "bd*16"
  # pF "azim" saw
  # pF "alt" (slow 2 saw)
  # pF "radius" (slow 3 saw)

.. but can't hear any panning.

Here's what I'm trying to do, binaural panning..

FoaTransform.ar(FoaEncode.ar(signals.sum*mul, ~encoder), 'push', pi/4, pan)

With this at startup:

~encoder = FoaEncoderMatrix.newOmni
~decoder = FoaDecoderKernel.newCIPIC

I can't get it working with superdirt though, even if I directly replace the call to DirtPanBalance2 by editing SuperDirtUGens.sc ..

Have you rebuilt the synthdefs?

perhaps try ~dirt.loadSynthDefs

Thanks! That gets me a bit further. Here's what I'm trying:

s.options.memSize = 8192 * 64 * 2; // increase this if you get "alloc failed" messages
s.boot

~dirt = SuperDirt(2, s); // two output channels, increase if you want to pan across more channels
~encoder = FoaEncoderMatrix.newOmni
~decoder = FoaDecoderKernel.newCIPIC


DirtPan.defaultPanningFunction = #{ | signals, numChannels, pan, mul |
	FoaTransform.ar(FoaEncode.ar(signals.sum*mul, ~encoder), 'push', pi/4, pan)
};
~dirt.loadSoundFiles;

~dirt.start(57120, [0, 0, 0, 0, 0, 0, 0, 0, 0]);

~dirt.loadSynthDefs

The final line gives an error.. It seems signals is nil somehow?

FoaRotate input 0 is not audio rate:  nil nil
 ARGS:
   in: nil Nil
   angle: an OutputProxy OutputProxy
   mul: an OutputProxy OutputProxy
   add: an OutputProxy OutputProxy
   4: an UnaryOpUGen UnaryOpUGen
SynthDef dirt_sample_1_2 build failed
ERROR: FoaRotate input 0 is not audio rate:  nil nil

Hi @micah-frank-studio, did you get further with this?

During lockdown I'm interested in being able to do streamed binaural performances, to have more presence for people listening on headphones.

I tried for several days and got it working but the radians were off. It was a steep learning curve for me not having ever learned supercollider. I was under the gun and had to move on so I ended developing a 5th order ambisonic system in Csound: https://vimeo.com/367455399

Would love to help out with this if I can free up some time...

@micah-frank-studio sorry, I didn't know you would have needed further help!

@yaxu

the last ~dirt.loadSynthDefs is not needed anymore when you have set your defaultPanningFunction already.

I did get the AST stuff working, but it was a bit underwhelming. Perhaps I was still doing something wrong, or maybe the headphones I used weren't good enough.

I had a similar experience but I thought some of my radians were off and thus the phase relationships. Even with mediocre headphones you should hear something "spatial". So maybe it's the toolkit.

On Mon, Jun 8, 2020 at 11:10 AM Alex McLean [email protected] wrote:

I did get the AST stuff working, but it was a bit underwhelming. Perhaps I was still doing something wrong, or maybe the headphones I used weren't good enough.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/musikinformatik/SuperDirt/issues/141#issuecomment-640691212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARBWJP6AADBSJFHTZM2DD3RVT5PDANCNFSM4ILYSQ2A .

The toolkit is done by people who are really experts as far as I can tell. Probably one needs to know a bit more in order to use it properly? One could ask them for advice.

I tried with slightly better headphones (over-ear headset designed for gaming, rather than some earbuds that probably came free with an android phone), and the results seem much better.

Do you folks know about the Sursound mailing list?

https://www.ambisonic.net/sursound.html

Might be a good place to get advice.

I am trying this from @munshkr:

(
e = FoaEncoderMatrix.newOmni;
d = FoaDecoderKernel.newCIPIC(12);
)

(
DirtPan.defaultPanningFunction = { |signals, numChannels, pan, mul|
  var sig = FoaEncode.ar(signals.sum * mul, e);
  // angle=pi/2 -> push to plane wave
  sig = FoaPush.ar(sig, angle: pi/2, theta: pan * 2*pi);
  FoaDecode.ar(sig, d);
};

~dirt.loadSynthDefs;
)

The ~dirt.loadSynthDefs is needed, otherwise it's the usual linear pan from left to right, even if I run this between ~dirt = SuperDirt(2, s) and ~dirt.start. I can then hear some spatialisation, however it sounds a bit funny, heavy on the right ear. I wonder if things are somehow getting stacked up so both panning functions are applied?

Hm maybe I'm imagining it, these sound OK:

d1 $ sound "sd*16"
   # pan (slow 8 saw)

d1 $ weave 16 (pan saw) [sound "sd:15(5,16)",
                         sound "clap:4(3,8)",
                         sound "~ sd:3",
                         sound "jvbass(3,8,2)"
                        ]

Still it feels like a kind of ellipse through my head (up-down and left right, rather than forward-back), rather than sounds going around me :)

Maybe the scaling of the pan parameter is not right? The parameter receives a range [-1..1] (which correspond to tidal's [0...1]), and the correct input range of theta in FoaPush is "Azimuth, in radians.".

So perhaps it should say: sig = FoaPush.ar(sig, angle: pi/2, theta: pan * pi); ?

Yes that seems much better thanks!

Maybe the scaling of the pan parameter is not right? The parameter receives a range [-1..1] (which correspond to tidal's [0...1]), and the correct input range of theta in FoaPush is "Azimuth, in radians.".

So perhaps it should say: sig = FoaPush.ar(sig, angle: pi/2, theta: pan * pi); ?

Right. I figured the azimuth went from 0 to 2pi, but after reading some examples in the documentation I found out that it ranged from -pi to pi. I thought that the pan argument ranged from 0-1 too. Thanks!

Excellent. For a more complete implementation, it would be good to differentiate between the different number of input channels, like in the defaultPanningFunction. It is not an easy question how to map many to many, but it is good to have one possible implementation there that can be hacked for different purposes.

Thanks @munshkr (and @yaxu) for pointing me to this approach with the defaultPanningFunction. It works well with the omni encoder, which is a matrix encoder (followed by push). But if you use one of the ATK kernel encoders (e.g. frequency diffusion or spreader) you will hear some audible glitches on short notes, which I believe is because the panning function and/or the note gets cut off before it's done...or something similar. I assume the kernel encoders work in frequency domain (FFT), so they work with 512, 1024 or 2048-frame chunks, and mustn't get cut off early. (at least, I suspect that's the reason for the subtle but audible glitches -- I'm not familiar enough with the whole signal chain with SuperDirt, orbits, etc.)

It would be nice to have the SuperDirt panning function only do encoding to B-format (ambisonic), then just have a single decoder at the end (whether per-orbit or on the whole SC output). (It would also be better for performance, if you have lots of fast notes!) This is the approach I've been taking when experimenting with this recently -- slapping a FoaDecode at the end of all SC output (after the main audio Group) and assuming the SC output channels (0-3) are encoded in B-format. You can see this and some other of my recent experiments here: https://github.com/totalgee/binaural-livecoding

Thanks @totalgee !

I have been hitting cpu problems hard with high polyphony, I guess because the panning is calculated separately for each individual sound event. I guess your approach will be a big improvement as you suggest, as the decoding is then only done once. Great! Thanks for sharing your code, I'll have a look.

I am due to make a composition for a multichannel system, but don't know too much about this. I guess it's possible to compose something in four channel surround, and save the result in this 'b format' somehow for sending to the multichannel lab? Due to covid19 restrictions I won't be able to go there myself for a while.

Yes, if you record (e.g. from SC) the output of the four (in the case of first-order ambisonics) B-format channels, you can then use that as a "golden master" from which it would be possible to render/decode for stereo, 5.1, 7.1, binaural stereo, arbitrary channel/speaker layouts, etc.

(In case you don't know), with higher-order ambisonics (e.g. 2nd-5th) you use more audio channels and processing power, but you gain spatial accuracy/precision. I've also done experiments with the SC-HOA Quark in SuperCollider (not shown in the repo above). I found it a bit more complicated to get working compared to ATK, but I did find the results more "precise" -- I found 3rd order a good compromise of not too heavy CPU usage (requires 16 channels vs 4 for its "B-format") but more precise spatial localization. In that case, I was using the SC-HOA Quark for encoding and transformation, but a VST plugin (from IEM) to do the higher-order decoding, because I found it took less CPU and gave similar or nicer results (for my ears) than the solution included with SC-HOA. I've mostly used this stuff for VR purposes; with head tracking it really comes into its own! Also, I know Joseph and the ATK gang have been working for a very long time on a higher-order version of ATK...but it's still not available (as a release for SC, at any rate).

This sounds vey nice.

Just in case you don't know anyhow: if you want to add a constant processing stage at the end of scsynth or superdirt, there are these options:

• for scsynth, you can use the tree instance variable: s.tree = { <put your synths here, they willl be restarted after someone stops everything with cmd-period> }
• for superdirt, you could add a global effect that comes before the monitor. Or, for that matter, you could just change the monitor synthdef (\dirt_monitor). There is an explanation here: https://github.com/musikinformatik/SuperDirt/blob/develop/hacks/adding-global-effects.scd