[ossia-max] feature request: test to measure "time to interactive"

[evanmtp]

I've been testing out various nightlies of ossia-max over the past while, and sometimes there seems to be some noticeable fluctuation in how long ossia takes to load for the first time. For example, as of f0ab85d, the first time I add any ossia-max object (ossia.parameter, ossia.remote, etc.) to a patcher, Max hangs for about 22 seconds.

I wrote a patch that could serve as the basis for a test:

ossia_tti_test.zip

It bangs an [ossia.parameter] as soon as it loads and uses its output to stop a [timer]. This gives a value in ms for "time to interactive" (TTI). It's important that the ossia.parameter is the first object created within the Max session, as (so far as I understand) loading the library happens when the first ossia object is instantiated.

Does it make sense to decide on a maximum acceptable load time for ossia-max and use this test to warn if it exceeds that? Even accounting for different hardware configs (and I don't know what sort of system GitHub/the CI tools are using to run unit tests), can we decide on a value that's just too slow?

I would have built this into a proper unit test, but there's currently a bug with ossia.test-logger now - please see #775 .

[jcelerier]

hm, here it seems to load in 23 ms on windows, which seems fairly reasonable...

[jcelerier]

you are really getting 22s of wait there ?

[jcelerier]

tried on mac too, it takes ~100ms which is slower but not shocking either

[evanmtp]

I just tested on macOS and I'm also getting ~100ms load time there. On Windows, unfortunately, it's still like this for me:

https://user-images.githubusercontent.com/5342408/160052572-e013de89-8b91-4621-9139-3c7df8c26c3c.mp4

Following your instructions in #765, I ran Max via windbg and paused execution right after creating the [ossia.parameter]. Here's the screenshot:

I noticed that this line was being printed in the windbg Command pane while Max was hanging:

dnssd_clientstub ConnectToServer: connect()-> No of tries: 1dnssd_clientstub ConnectToServer: connect()-> No of tries: 2dnssd_clientstub ConnectToServer: connect()-> No of tries: 3dnssd_clientstub ConnectToServer: connect()-> No of tries: 1dnssd_clientstub ConnectToServer: connect()-> No of tries: 2dnssd_clientstub ConnectToServer: connect()-> No of tries: 3

and after Max stopped hanging, these lines occured:

(491c.3ff0): C++ EH exception - code e06d7363 (first chance) (491c.3ff0): C++ EH exception - code e06d7363 (first chance)

[jcelerier]

Thanks, that's really useful ! The dnssd thing really hints towards a network issue (it's the "technical" name of Bonjour). I'll maje you a build without it so that we can confirm if it's the culprit

On Fri, 25 Mar 2022, 05:13 Evan Montpellier, @.***> wrote:

I just tested on macOS and I'm also getting ~100ms load time there. On Windows, unfortunately, it's still like this for me:

https://user-images.githubusercontent.com/5342408/160052572-e013de89-8b91-4621-9139-3c7df8c26c3c.mp4

Following your instructions in #765 https://github.com/ossia/libossia/issues/765, I ran Max via windbg and paused execution right after creating the [ossia.parameter]. Here's the screenshot:

[image: ossia_network_hang_01] https://user-images.githubusercontent.com/5342408/160052753-93aaa442-f067-40f3-904e-42a47b32c702.png

I noticed that this line was being printed in the windbg Command pane while Max was hanging:

dnssd_clientstub ConnectToServer: connect()-> No of tries: 1dnssd_clientstub ConnectToServer: connect()-> No of tries: 2dnssd_clientstub ConnectToServer: connect()-> No of tries: 3dnssd_clientstub ConnectToServer: connect()-> No of tries: 1dnssd_clientstub ConnectToServer: connect()-> No of tries: 2dnssd_clientstub ConnectToServer: connect()-> No of tries: 3

and after Max stopped hanging, these lines occured:

(491c.3ff0): C++ EH exception - code e06d7363 (first chance) (491c.3ff0): C++ EH exception - code e06d7363 (first chance)

— Reply to this email directly, view it on GitHub https://github.com/ossia/libossia/issues/776#issuecomment-1078635516, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVE56TPQVSZM4MIIPEZSZTVBU4N5ANCNFSM5RMMKXUQ . You are receiving this because you were assigned.Message ID: @.***>

[jcelerier]

Hummmm by any chance did you maybe not install Bonjour on windows ? https://support.apple.com/kb/dl999?locale=en_US

[evanmtp]

Eek, apparently not - I just installed it, and the hang is gone. Good catch!

Is it possible for ossia-max to check if Bonjour is installed when it loads? It would be great if it could avoid the hang and warn the user that Bonjour is required for network communication. Probably best if this doesn't prevent Max from loading ossia objects altogether, though - there might be some people who don't want to install Bonjour but still want to use ossia-max.

[jcelerier]

Is it possible for ossia-max to check if Bonjour is installed when it loads?

yes, that's definitely something we should do

[evanmtp]

Cool! I'm glad we got to the bottom of this specific issue.

Going back to the broader topic of the thread, what do you think about having a TTI automated test, in terms of both usefulness and feasibility?

For context: at a certain point, we at TML noticed that Jamoma had started to have inordinately long load/close times, to such a degree that it became very risky to put it onstage in performance/installation contexts (e.g. having a giant transient go through a PA system when DSP locks up, followed by by 4 minutes of silence while waiting for Max to close and reopen, all in front of an audience...). This is what motivated us to go looking for other options and what led us to ossia in the first place! Having some way of making sure that load times don't get out of hand due to some regression seems very important to me.

[avilleret]

Computing loading time is definitely a useful metric to detect performance regression. But for now ossia-max unit tests are not executed on the CI and I think we should not rely on GitHub VM to do so. Because 1) it's hard to install max on headless system, 2) GitHub VM have very high performance variation from one build to another. Instead we should have our own machine to run those tests. And we should only compare test results made on the same machine. I do have an old macmini at home and also some vm. I'll try to setup one of them as a GitHub action runner and run the tests on it.

[evanmtp]

That would be great, @avilleret !

I know that I've brought up variations of this issue in the past, but I'm currently working with @navid on a concert patch and load time has become very slow as we've added more ossia objects. It looks like we have 2259 nodes in the patch (based on passing the output of ossia.explorer to a t b and then counter). It takes upwards of three minutes for the patch to load, which is quite risky for a performance patcher.

@villeret and @jcelerier , if you would be willing to look at it, I can share the patcher later, along with its dependencies. If there's anything that can be done to mitigate this issue of load time increasing drastically as more ossia objects are added, that would be great. Also, if we're doing something in the patch that's causing this, and there's another approach that we're missing, that would be good to know.

[jcelerier]

@evanmtp yes, please do share the patch! 2250 nodes should not be slow at all