Add macOS minibrowser support to 'run-benchmark'

[lukewarlow]

18164cb581d24c8512875f75cefb3250c9f418c3

Add macOS minibrowser support to 'run-benchmark'
https://bugs.webkit.org/show_bug.cgi?id=275617

Reviewed by NOBODY (OOPS!).

Adds a new osx_minibrwser_driver.py to 'run-benchmark' command.

* Tools/Scripts/webkitpy/benchmark_runner/browser_driver/osx_minibrowser_driver.py: Added.
(OSXMiniBrowserDriver):
(OSXMiniBrowserDriver.launch_args_with_url):
(OSXMiniBrowserDriver.launch_url):
(OSXMiniBrowserDriver.launch_driver):
(OSXMiniBrowserDriver.set_binary_location_impl):

https://github.com/WebKit/WebKit/commit/18164cb581d24c8512875f75cefb3250c9f418c3

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	✅ 🧪 wincairo-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
✅ 🧪 webkitpy	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2
	✅ 🛠 tv
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

[webkit-early-warning-system]

EWS run on current version of this PR (hash https://github.com/WebKit/WebKit/commit/18164cb581d24c8512875f75cefb3250c9f418c3)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	✅ 🧪 wincairo-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
✅ 🧪 webkitpy	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2
	✅ 🛠 tv
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

[weinig]

Assuming the name is not something that is required, I would avoid "osx" and use "macOS" instead (as macOS is no longer called that).

[lukewarlow]

The name is required to be osx as far as I know. The script lets you pick an os (Linux or osx) and then a browser and the valid browser is based on the OS prefix. So if I did macOS I'd have to update the tooling. I agree it should all be renamed but it's not something for this PR imo.

[aproskuryakov]

What is the motivation for this? I know that this has been brought up before as "measure WebKit performance without overhead from Safari features", but the interactions are much more complicated than "overhead", so we haven't figured out what we'd do with such data points.

[justinmichaud]

@aproskuryakov Safari for WebKit development is frequently broken, and it is also nice to get this data on linux

[aproskuryakov]

I think that SafariForWebKitDevelopment is sometimes broken without SIP, which is unfortunate but not a blocker. I don't see why other breakage shouldn't be fixed when running into it – seems better than getting misleading performance numbers.

it is also nice to get this data on linux

This PR is about macOS though?

[lukewarlow]

seems better than getting misleading performance numbers.

Can I ask why they'd be misleading? You'd only be able to compare to other minibrowser numbers (with the same build parameters such as debug vs release) but the numbers would still be comparable across equivalent builds right?

[lukewarlow]

~~I think I've managed to get it working with safari locally~~ so for now I'll close this. Though I still personally think it would be a useful addition

Edit: seems like it's still just using system safari.

[aproskuryakov]

Can I ask why they'd be misleading? You'd only be able to compare to other minibrowser numbers (with the same build parameters such as debug vs release) but the numbers would still be comparable across equivalent builds right?

I don't think that we can even always expect movement in the same direction - a MiniBrowser progression could be a Safari regression, and vice versa. Given that MiniBrowser itself is not a target for optimization, this makes the numbers fairly useless, unless there's a scenario that I'm not thinking of.

[justinmichaud]

While Minibrowser may not be an optimization target for Apple, being able to get performance numbers at-desk is important if we want to avoid needing a round-trip with Apple engineers. This way, we can only ask for perf tasks for things that we are already somewhat confident aren't regressions.

Since folks outside Apple don't have any of the internal bits or PGO profiles, no performance results under say, 7%, could ever be useful to Apple anyway. MiniBrowser results still seem more useful than linux results, for example.

[aproskuryakov]

I think that this discussion is really dependent on "SafariForWebKitDevelopment isn't working for us", and it would be preferable to figure that out if we can?