Level playing field

[ipjohnson]

trafficstars

With the changes introduced with issue #71 we attempted to level the playing field and come up with a more real world test. This give a speed advantage to containers that have less features.

This wasn't a concern before as the top containers all competed in all the benchmarks (minus the child container). Now that abioc is the top container it competes in less than half the benchmarks and has a real advantage.

My suggestion is to add a method for each adapter that is called at the end of warm up to resolve some dummy services to level the play field.

So for example Stashbox, Grace and Autofac would resolve 0 dummy registration as they compete in all, LightInject, DryIoC would resolve 3 dummy registration, abioc would resolve 18 (6 benchmarks x 3) and so forth.

[ipjohnson]

Also we'll want to make sure each container has roughly the same number of registrations not just resolving extra registrations.

@danielpalme I'd like to put together a PR for this if you'd accept it.

[dadhi]

hi, I would rather add NoContainer X 20 time to missing bmarks of other iocs, plus fill them in grey on charts.

[dadhi]

Sorry, not "rather" but combine this with above suggestion

[danielpalme]

I'm not sure, which approach is better here. But adding NoContainer X 20 seems a bit random and does not correlate with the container's performance at all. But as abcioc is still under development, its performance will most likely decrease in the future.

[ipjohnson]

@danielpalme I thought about it a bit more and I'm torn. On one hand I'm a firm believer in having benchmarks that are equal and fair to all. On the other hand as it is still in development it has few features and people are probably not going to use just for it's minor speed improvements when it doesn't support the platform they are developing on (UWP, Xarmain, MVC, etc)

What if you just put a note saying the performance isn't exactly apples to apple for abioc and revisit the topic in a couple months to see what type of progress it's made?

The reality is I probably have other things I should be working on and while I'd like it to be a level playing field at the end of the day it's really doesn't matter :)

[danielpalme]

Let's just keep this issue open and have a look at it in a few weeks. I believe there are more important things to do.

[ENikS]

As discussed in #77 it is apparent that current registration strategy creates a loophole for containers to cheat on these benchmarks. I would like to offer a solution which would eliminate it.

Instead of registering all required types, adapter should provide override to these three methods:

object BeginRegistration(); // Returns builder
void RegisterType(object builder, Type registeredType, Type mappedTo, string name = null);
void RegisterInstance(object builder, Type registeredType, object instance, string name = null);
IBenchmarkServiceLocator EndRegistration(object builder);

// Interception should be tested as a separate container
IBenchmarkServiceLocator RegisterInterception(...); // Not sure what to pass here

// Asp.Net should be tested as a separate container
IServiceProvider GetServiceProvider(IServiceCollection services);

The framework itself should call RegisterType and RegisterInstance to initialize container and once done run tests. This will also allow to expand tests without need to modify adapters.

Optimizations and implementation details of interception and asp.net might affect container's performance so these should be tested separately.