Poor REST performance vs. eXist

[IanDavey]

I have been benchmarking different types of requests to the RESTful API for both Fusion and eXist, using the latest USLM versions of titles 1 (a short one) and 42 (the longest one) of the US Code. Each of these tests had 100 repetitions, and both eXist and Fusion were identically configured on the same VM.

First, I tested a simple PUT of a document to /exist/rest/db/test:

usc01.xml eXist:
    Minimum: 0.43300747871398926 s
    Maximum: 1.388643503189087 s
    Mean:    0.6491883111000061 s
    Median:  0.6234605312347412 s
    Stddev:  0.15249426666857233 s
usc01.xml Fusion:
    Minimum: 0.5359818935394287 s
    Maximum: 2.7659928798675537 s
    Mean:    1.2113053154945375 s
    Median:  1.1449862718582153 s
    Stddev:  0.47974903573925054 s
usc42.xml eXist:
    Minimum: 35.580822229385376 s
    Maximum: 54.87896108627319 s
    Mean:    41.51840656995773 s
    Median:  41.32154309749603 s
    Stddev:  4.195144231762311 s
usc42.xml Fusion:
    Minimum: 188.74798727035522 s
    Maximum: 1077.4066643714905 s
    Mean:    404.10182027816774 s
    Median:  320.5594769716263 s
    Stddev:  191.0485587590224 s

Next, I tested GET:

usc01.xml eXist:
    Minimum: 0.03500247001647949 s
    Maximum: 0.2610175609588623 s
    Mean:    0.05440183639526367 s
    Median:  0.0410001277923584 s
    Stddev:  0.04378882484633538 s
usc01.xml Fusion:
    Minimum: 0.5689961910247803 s
    Maximum: 0.7550196647644043 s
    Mean:    0.6137096118927002 s
    Median:  0.6049911975860596 s
    Stddev:  0.030363894050856537 s
usc42.xml eXist:
    Minimum: 13.213871717453003 s
    Maximum: 15.583616971969604 s
    Mean:    13.67682737827301 s
    Median:  13.544020891189575 s
    Stddev:  0.40090442662817194 s
usc42.xml Fusion:
    Minimum: 264.0210030078888 s
    Maximum: 293.1509966850281 s
    Mean:    272.6948435425758 s
    Median:  269.29898858070374 s
    Stddev:  7.9907570945070505 s

Then, I POSTed some XQuery to get a specific section of a document:

1 U.S.C. 1 eXist:
    Minimum: 0.004994392395019531 s
    Maximum: 0.011027336120605469 s
    Mean:    0.006702215671539307 s
    Median:  0.0060079097747802734 s
    Stddev:  0.0012504552254434967 s
1 U.S.C. 1 Fusion:
    Minimum: 0.00799870491027832 s
    Maximum: 0.03599071502685547 s
    Mean:    0.010678362846374512 s
    Median:  0.009988903999328613 s
    Stddev:  0.0033822034856866566 s
42 U.S.C. 2000e eXist:
    Minimum: 0.012010574340820312 s
    Maximum: 0.04998445510864258 s
    Mean:    0.016210291385650635 s
    Median:  0.014001250267028809 s
    Stddev:  0.005966727038671187 s
42 U.S.C. 2000e Fusion:
    Minimum: 0.02099442481994629 s
    Maximum: 0.06598377227783203 s
    Mean:    0.026210627555847167 s
    Median:  0.02399766445159912 s
    Stddev:  0.006877559298661334 s

(the XQuery I used was an adapted version of a giant block we're using for a client that does a bunch of pre- and post-processing, but the most relevant piece is //*[@identifier=$identifier], and this is an indexed attribute)

Finally, I ran a DELETE (PUTting the document back between repetitions, but only timing the DELETE):

1 U.S.C. eXist:
    Minimum: 0.4656083583831787 s
    Maximum: 1.5618107318878174 s
    Mean:    0.6475286889076233 s
    Median:  0.5726385116577148 s
    Stddev:  0.2086682498874921 s
1 U.S.C. Fusion:
    Minimum: 2.4692959785461426 s
    Maximum: 4.827873945236206 s
    Mean:    3.485269944667816 s
    Median:  3.424243688583374 s
    Stddev:  0.5818232474143926 s
42 U.S.C. eXist:
    Minimum: 18.95114755630493 s
    Maximum: 26.387588024139404 s
    Mean:    21.3156515455246 s
    Median:  21.073147296905518 s
    Stddev:  1.509445739774873 s
42 U.S.C. Fusion:
    Minimum: 1957.4530036449432 s
    Maximum: 2816.1365151405334 s
    Mean:    2314.854492249489 s
    Median:  2259.181742668152 s
    Stddev:  224.80505757124126 s

It appears that Fusion's running time across the board grows faster than eXist with respect to document size. As several of Xcential's use cases involve handling large-ish documents, as a result we currently can't recommend Fusion over eXist to clients (which is a unfortunate, because in other POST XQuery tests, such as generating a small top-level outline for all USC titles, Fusion is faster).