Filter mappings with URL first

[mbeczek]

Hi! Same case like here https://github.com/tomakehurst/wiremock/issues/1152 , but other problem.

I found that WireMock is checking every mapping with body patterns and other matchers, no matter that URL is not matching - I assume that is the most important part of request. Even if I have mappings done well for single API I'm checking, I still verify other mappings - with other URLs and, what's worse, with very bad body patterns.

WireMock version: 2.18.0

Thanks!

[tomakehurst]

This is already on my private list of things to consider, along with some similar design options like organising stubs into a tree in order to avoid a linear search every time.

Probably not going to happen for a while though - I think this is a v3.0+ thing.

[htmldoug]

I've implemented both of those InMemoryStubMappings optimizations locally. The speedups are significant.

Short-circuiting on isExactMatch is pretty trivial to implement for the gains. I just replaced this with schemeMatches().isExactMatch && hostMatches().isExactMatch && ....

The radix tree was tricker. I'm splitting on "/" and truncating at the first sign of regex. All tree nodes maintain a sorted set of stubs, and the whole matching branch gets combined with Iterators.mergeSorted. There were also thread-safety considerations to deal with on add/edit/removeMapping.

Over 100k stubs with colliding paths (1,000x faster):

bench / Jmh / run -t1 -p size=100000 .*SamePathDifferentHeadersStubMappingsBench
[info] Benchmark                                           (size)  (stubs)   Mode  Cnt    Score    Error  Units
[info] SamePathDifferentHeadersStubMappingsBench.serveFor  100000  default  thrpt   20    0.217 ±  0.004  ops/s
[info] SamePathDifferentHeadersStubMappingsBench.serveFor  100000     fast  thrpt   20  239.631 ± 41.280  ops/s

With distinct paths (100,000x faster):

[info] Benchmark                                           (size)  (stubs)   Mode  Cnt      Score     Error  Units
[info] DistinctStubMappingsBench.serveFor                  100000     fast  thrpt   20  24970.602 ± 221.665  ops/s

Last compared against 2.26.3.

[htmldoug]

FYI, I also tested implementing StubMappings over https://github.com/npgall/cqengine which greatly simplifies the implementation. It seems to have a bit more fixed overhead than the current one. It's slower at low numbers of stubs, but scales nicely to high numbers of stubs. It's not as fast as my hand-rolled implementation, particularly for larger result sets that it has to sort, but it's much less complex. Might be worth exploring.

Relevant bits:

val Method = attribute((_: StubMapping).getRequest.getMethod.getExpected)
val PathPrefix = attribute((_: StubMapping).getRequest.getUrlMatcher.nonRegexPrefix) // "/" for empty
val Priority = attribute((stub: StubMapping) => Option(stub.getPriority).getOrElse(Integer.valueOf(5)))
val InsertionIndex = attribute((stub: StubMapping) => java.lang.Long.valueOf(stub.getInsertionIndex))

val db = new ObjectLockingIndexedCollection[StubMapping]()
db.addIndex(HashIndex.onAttribute(Method))
db.addIndex(InvertedRadixTreeIndex.onAttribute(PathPrefix))

// serveFor
val query = and(
  in(Method, request.getMethod.getExpected, "ANY"),
  isPrefixOf(PathPrefix, request.getUrl.path)
)
val options = queryOptions(orderBy(ascending(Priority), descending(InsertionIndex)))
db.retrieve(query, options).iterator()

[tomakehurst]

Hi @htmldoug, thanks for all the contributions you're making at the moment and apologies for not having reviewed most of them yet. I'll be less underwater from the middle of next week hopefully so will try to catch up.

In the meantime, we've just started Discord server for contributors and maintainers. Would you like an invite?

[htmldoug]

Thanks @tomakehurst. Sure, I'd accept a discord invite. Doug Roper#7092.