feat(server): spawn task sooner in listenloop + fast accept

Open pankgeorg opened this issue 2 years ago • 1 comments

So, this is a bit speculative, but I'm very very concerned about the backlog getting big; I think it's always a good idea to get through as many accepts as possible, before spawning the tasks to handle the connections. This builds on https://github.com/JuliaWeb/HTTP.jl/pull/1102 and adds a acceptmany function that returns a vector of accepted connections, if they are available.

I've run some benchmarks (this commit) on this and it seems that it helps for the benchmarks that are connection intensive:

see latency-average for plaintext and json (the db related queries also benefit from async query send)

+-----------------------------------------------+
|       Type: fortune, Result: latencyAvg       |
+-------------------+---------+-----------------+
| concurrencyLevels | http-jl | http-jl-patched |
+-------------------+---------+-----------------+
|                16 |  4.37ms |          4.41ms |
|                32 |  4.53ms |          4.35ms |
|                64 |  5.43ms |          5.71ms |
|               128 |  6.72ms |          6.05ms |
|               256 | 26.63ms |         19.72ms |
|               512 | 41.97ms |         34.58ms |
+-------------------+---------+-----------------+

+--------------------------------------------------------+
|          Type: plaintext, Result: latencyAvg           |
+---------------------------+----------+-----------------+
| pipelineConcurrencyLevels |  http-jl | http-jl-patched |
+---------------------------+----------+-----------------+
|                       256 | 121.11ms |         69.99ms |
|                      1024 | 253.44ms |        130.26ms |
|                      4096 | 210.46ms |        158.88ms |
|                     16384 | 628.35ms |        526.72ms |
+---------------------------+----------+-----------------+

+-----------------------------------------------+
|          Type: db, Result: latencyAvg         |
+-------------------+---------+-----------------+
| concurrencyLevels | http-jl | http-jl-patched |
+-------------------+---------+-----------------+
|                16 |  2.90ms |          3.56ms |
|                32 |  3.69ms |          3.54ms |
|                64 |  4.85ms |          4.70ms |
|               128 |  5.28ms |          5.35ms |
|               256 | 26.99ms |         19.78ms |
|               512 | 37.59ms |         32.57ms |
+-------------------+---------+-----------------+

+------------------------------------------------+
|        Type: update, Result: latencyAvg        |
+-------------------+----------+-----------------+
| concurrencyLevels |  http-jl | http-jl-patched |
+-------------------+----------+-----------------+
|                16 |  47.18ms |         45.10ms |
|                32 | 217.47ms |        219.50ms |
|                64 | 434.74ms |        412.98ms |
|               128 | 622.48ms |        604.69ms |
|               256 | 865.47ms |        816.59ms |
+-------------------+----------+-----------------+

+------------------------------------------------+
|         Type: json, Result: latencyAvg         |
+-------------------+----------+-----------------+
| concurrencyLevels |  http-jl | http-jl-patched |
+-------------------+----------+-----------------+
|                16 | 163.85us |        153.34us |
|                32 | 163.61us |        150.35us |
|                64 |   1.40ms |        553.35us |
|               128 |   1.50ms |        486.40us |
|               256 |   6.25ms |          4.13ms |
|               512 |   7.99ms |          3.57ms |
+-------------------+----------+-----------------+

+---------------------------------------------+
|       Type: query, Result: latencyAvg       |
+----------------+----------+-----------------+
| queryIntervals |  http-jl | http-jl-patched |
+----------------+----------+-----------------+
|              1 |  35.18ms |         31.81ms |
|              5 | 132.25ms |        120.80ms |
|             10 | 257.01ms |        234.04ms |
|             15 | 373.05ms |        338.24ms |
|             20 | 486.86ms |        459.46ms |
+----------------+----------+-----------------+

+-----------------------------------------------+
|      Type: fortune, Result: totalRequests     |
+-------------------+---------+-----------------+
| concurrencyLevels | http-jl | http-jl-patched |
+-------------------+---------+-----------------+
|                16 |  54,946 |          54,549 |
|                32 | 106,102 |         110,678 |
|                64 | 178,001 |         171,313 |
|               128 | 179,736 |         199,352 |
|               256 | 219,430 |         241,545 |
|               512 | 234,065 |         248,983 |
+-------------------+---------+-----------------+

+----------------------------------------------------------+
|          Type: plaintext, Result: totalRequests          |
+---------------------------+------------+-----------------+
| pipelineConcurrencyLevels |    http-jl | http-jl-patched |
+---------------------------+------------+-----------------+
|                       256 |  9,638,599 |      10,832,911 |
|                      1024 | 10,132,321 |      12,119,105 |
|                      4096 |  8,297,343 |       8,745,954 |
|                     16384 |  6,745,883 |       7,421,888 |
+---------------------------+------------+-----------------+

+-----------------------------------------------+
|        Type: db, Result: totalRequests        |
+-------------------+---------+-----------------+
| concurrencyLevels | http-jl | http-jl-patched |
+-------------------+---------+-----------------+
|                16 |  82,778 |          67,685 |
|                32 | 130,414 |         138,148 |
|                64 | 205,705 |         205,374 |
|               128 | 228,337 |         224,848 |
|               256 | 263,888 |         285,058 |
|               512 | 279,247 |         295,292 |
+-------------------+---------+-----------------+

+-----------------------------------------------+
|      Type: update, Result: totalRequests      |
+-------------------+---------+-----------------+
| concurrencyLevels | http-jl | http-jl-patched |
+-------------------+---------+-----------------+
|                16 | 164,456 |         167,064 |
|                32 |  33,999 |          34,727 |
|                64 |  17,020 |          17,632 |
|               128 |  11,639 |          11,875 |
|               256 |   8,472 |           8,757 |
+-------------------+---------+-----------------+

+-------------------------------------------------+
|        Type: json, Result: totalRequests        |
+-------------------+-----------+-----------------+
| concurrencyLevels |   http-jl | http-jl-patched |
+-------------------+-----------+-----------------+
|                16 | 1,572,458 |       1,684,229 |
|                32 | 3,150,079 |       3,470,513 |
|                64 | 4,912,047 |       5,626,121 |
|               128 | 5,854,543 |       5,812,078 |
|               256 | 7,094,927 |       7,763,330 |
|               512 | 7,132,833 |       8,553,204 |
+-------------------+-----------+-----------------+

+--------------------------------------------+
|     Type: query, Result: totalRequests     |
+----------------+---------+-----------------+
| queryIntervals | http-jl | http-jl-patched |
+----------------+---------+-----------------+
|              1 | 281,056 |         293,010 |
|              5 |  62,317 |          65,484 |
|             10 |  30,851 |          32,613 |
|             15 |  20,595 |          22,229 |
|             20 |  15,926 |          16,422 |
+----------------+---------+-----------------+

Sep 03 '23 14:09 pankgeorg

Codecov Report

Merging #1103 (811b6b3) into master (8f35185) will increase coverage by 0.04%. The diff coverage is 89.06%.

@@            Coverage Diff             @@
##           master    #1103      +/-   ##
==========================================
+ Coverage   82.34%   82.38%   +0.04%     
==========================================
  Files          32       33       +1     
  Lines        3042     3089      +47     
==========================================
+ Hits         2505     2545      +40     
- Misses        537      544       +7

Files Changed	Coverage Δ
src/HTTP.jl	`67.05% <ø> (ø)`
src/WebSockets.jl	`87.50% <ø> (ø)`
src/Servers.jl	`79.90% <88.23%> (-0.31%)`	:arrow_down:
src/accept.jl	`90.00% <90.00%> (ø)`

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Sep 03 '23 14:09 codecov-commenter