FrameworkBenchmarks
FrameworkBenchmarks copied to clipboard
TechEmpower
TLS/SSL test type(s)
This issue is for planning a new test type that exercises TLS (SSL), which was suggested as test type 10 on issue #133. I recommend reviewing the comment thread for #133 since it contains several prior thoughts concerning a TLS/SSL test type that remain applicable. Though I am not manually copying those comments over here, please feel free to reiterate points raised there.
Add any comments and questions that you believe can help us reach a consensus on reasonable requirements for this new test type.
Not all participants will agree that a TLS test is warranted in this project. Historically, we have not focused on the performance of web servers, intending our project to be about web application frameworks. However, our project is about measuring the high-water mark of performance using real-world web application technology stacks. And from that perspective, including a TLS test type is reasonable as long as there are some stacks for which TLS would be included "in-stack." And I know that is satisfied in the real-world because many stacks include nginx or another mainstream web server that has robust TLS capability.
Perhaps it is important to remember that, to whatever degree it is possible, we want the test implementations in this project to represent canonical usage of the platforms and frameworks we exercise. Many platforms and frameworks are of the opinion that TLS is an external concern to be handled by a front-end web server, load balancer, or other external system or service. That is perfectly fine and acceptable. Such platforms and frameworks may elect to bypass implementing this TLS test type. All test types in this project are optional, and omission should not be considered a failing or even evidence of lack of capability for a framework or platform. (Aside: We intend to eventually provide contributors a way to document the architectural approach that they have taken, explain the details of each test implementation they have provided, and in the case of omitting a test, explain why that test type is not applicable.)
I propose the following:
- This may be split into multiple test types since real-world TLS usage is applied to any and all request types. However, I am reluctant to duplicate many existing test types for a few reasons (e.g., additional test types increase the total runtime of the full suite). I tentatively propose TLS versions of two existing test types: Plaintext and Fortunes. Tentatively, TLS-Plaintext and TLS-Fortunes.
- If possible, I'd like to agree to a singular TLS cipher-suite configuration that is broadly supported and considered acceptable to a wide audience. If that is infeasible, I'd like for us to set some "minimum" acceptable level and illustrate the options that satisfy the minimum.
- If possible, I'd like to find and use a tool that validates TLS configuration that we can use in our suite's validation of implementations. Perhaps curl or similar can do this; I am not sure. But by way of comparison, I am imagining something that is approximately a command-line version of the SSL Labs tests.
Thoughts, questions, corrections, comments appreciated!
I think the openssl client is probably a good start for testing for the configuration of a server.
I agree that it is pointless replicating all tests for TLS as one or two like you have proposed will give enough data to provide a Delta between TLS and non TLS tests to allow for an overhead calculation.
I would say that AES is the most widely supported algo for the bulk ciper. Sha256 or higher for the hash shouldn't be controversial either. The main issue will be deciding on the certificate and initial key exchange.
The two tests should be enough to extrapolate from when comparing to non-TLS.
TLS should be 1.2 (1.3 isn't widely supported yet, < 1.1 is bad)
Cipher Suite should be encouraged to be a common one between frameworks; as there are dramatic differences in performances and varying them would be comparing the algorithm rather than the differing effects of using TLS.
Not sure what the cipher suite should be.
SSL Labs Cipher Suite (in order of preference) reports for various browsers
Chrome
TLS_GREASE_1A (0x1a1a) -
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) Forward Secrecy | 128
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) Forward Secrecy | 256
TLS_RSA_WITH_AES_128_GCM_SHA256 (0x9c) WEAK | 128
TLS_RSA_WITH_AES_256_GCM_SHA384 (0x9d) WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f) WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35) WEAK | 256
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa) WEAK | 112
Edge
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c) Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) Forward Secrecy | 128
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024) Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023) Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) Forward Secrecy | 128
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a) Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009) Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) Forward Secrecy | 128
TLS_RSA_WITH_AES_256_GCM_SHA384 (0x9d) WEAK | 256
TLS_RSA_WITH_AES_128_GCM_SHA256 (0x9c) WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA256 (0x3d) WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA256 (0x3c) WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35) WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f) WEAK | 128
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa) WEAK | 112
Firefox
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) Forward Secrecy | 128
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8) Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a) Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009) Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) Forward Secrecy | 256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33) Forward Secrecy | 128
TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39) Forward Secrecy | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f) WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35) WEAK | 256
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa) WEAK | 112
Top common is TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) ?
However, our project is about measuring the high-water mark of performance using real-world web application technology stacks. And from that perspective, including a TLS test type is reasonable as long as there are some stacks for which TLS would be included "in-stack." And I know that is satisfied in the real-world because many stacks include nginx or another mainstream web server that has robust TLS capability.
Would an "Is edge configuration" flag be sensible? TLS is valid in non-internet facing stacks; and there are already stacks in the tests that use for example ngnix as their front-end server; so they would presumably be edge/internet facing ready.
However, that can come with a performance disadvantage, so comparing non-internet facing stacks against internet facing stacks may be a little apples and oranges. While both fruit, one is more resistant to bruising?
The disadvantage I can see to that is it make be taken as approval that a particular stack is internet facing ready; while it may not be. So would at least need caveats; and may not be good for that reason?
@bhauer
If possible, I'd like to find and use a tool that validates TLS configuration that we can use in our suite's validation of implementations. Perhaps curl or similar can do this; I am not sure. But by way of comparison, I am imagining something that is approximately a command-line version of the SSL Labs tests.
I'm using this one: https://testssl.sh/, https://github.com/drwetter/testssl.sh I think it will tick every box: mainly shell script, non-exotic external dependencies, similar functionality to SSL Labs tests.
Off-topic: there is a similar command-line tool for SSH auditing https://github.com/arthepsy/ssh-audit It is not related to the testssl.sh.
Update: Here is a sample report for https://www.techempower.com/:
xyz@zyx:~/tmp/testssl.sh$ ./testssl.sh https://www.techempower.com/
###########################################################
testssl.sh 2.9dev from https://testssl.sh/dev/
(7e62dc3 2017-12-07 09:59:58 -- )
This program is free software. Distribution and
modification under GPLv2 permitted.
USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!
Please file bugs @ https://testssl.sh/bugs/
###########################################################
Using "OpenSSL 1.0.2-chacha (1.0.2i-dev)" [~183 ciphers]
on ######:./bin/openssl.Linux.x86_64
(built: "Jun 22 19:32:29 2016", platform: "linux-x86_64")
Start 2018-02-15 21:53:39 -->> 69.89.75.196:443 (www.techempower.com) <<--
rDNS (69.89.75.196): --
Service detected: HTTP
Testing protocols via sockets except SPDY+HTTP2
SSLv2 not offered (OK)
SSLv3 not offered (OK)
TLS 1 offered
TLS 1.1 offered
TLS 1.2 offered (OK)
TLS 1.3 not offered
SPDY/NPN http/1.1 (advertised)
HTTP2/ALPN http/1.1 (offered)
Testing ~standard cipher categories
NULL ciphers (no encryption) not offered (OK)
Anonymous NULL Ciphers (no authentication) not offered (OK)
Export ciphers (w/o ADH+NULL) not offered (OK)
LOW: 64 Bit + DES encryption (w/o export) not offered (OK)
Weak 128 Bit ciphers (SEED, IDEA, RC[2,4]) not offered (OK)
Triple DES Ciphers (Medium) offered
High encryption (AES+Camellia, no AEAD) offered (OK)
Strong encryption (AEAD ciphers) offered (OK)
Testing robust (perfect) forward secrecy, (P)FS -- omitting Null Authentication/Encryption, 3DES, RC4
PFS is offered (OK) ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA DHE-RSA-CAMELLIA256-SHA ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-SHA256
DHE-RSA-AES128-SHA DHE-RSA-CAMELLIA128-SHA
Elliptic curves offered: prime256v1
Testing server preferences
Has server cipher order? yes (OK)
Negotiated protocol TLSv1.2
Negotiated cipher ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Cipher order
TLSv1: ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA AES128-SHA AES256-SHA
DHE-RSA-CAMELLIA256-SHA CAMELLIA256-SHA DHE-RSA-CAMELLIA128-SHA CAMELLIA128-SHA DES-CBC3-SHA
TLSv1.1: ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA AES128-SHA AES256-SHA
DHE-RSA-CAMELLIA256-SHA CAMELLIA256-SHA DHE-RSA-CAMELLIA128-SHA CAMELLIA128-SHA DES-CBC3-SHA
TLSv1.2: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA256
DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA AES128-GCM-SHA256 AES256-GCM-SHA384 AES128-SHA256
AES256-SHA256 AES128-SHA AES256-SHA DHE-RSA-CAMELLIA256-SHA CAMELLIA256-SHA DHE-RSA-CAMELLIA128-SHA CAMELLIA128-SHA
DES-CBC3-SHA
Testing server defaults (Server Hello)
TLS extensions (standard) "server name/#0" "renegotiation info/#65281" "EC point formats/#11" "session ticket/#35"
"status request/#5" "heartbeat/#15" "next protocol/#13172"
"application layer protocol negotiation/#16"
Session Ticket RFC 5077 hint 86400 seconds, session tickets keys seems to be rotated < daily
SSL Session ID support yes
Session Resumption Tickets: yes, ID: yes
TLS clock skew Random values, no fingerprinting possible
Signature Algorithm SHA256 with RSA
Server key size RSA 4096 bits
Fingerprint / Serial SHA1 D6B3A1EF3FA6574D400E2700FD3601134355F060 / 5EB3A5F4DF31299DD34E5D135D8B4679
SHA256 F5E30EA7EA319419848E1F024566135244E223B50147118CE02A635D345672F8
Common Name (CN) *.techempower.com
subjectAltName (SAN) *.techempower.com techempower.com
Issuer COMODO RSA Domain Validation Secure Server CA (COMODO CA Limited from GB)
Trust (hostname) Ok via SAN wildcard and CN wildcard (same w/o SNI)
Chain of trust Ok
EV cert (experimental) no
Certificate Expiration 282 >= 60 days (2015-11-13 02:00 --> 2018-11-25 01:59 +0200)
# of certificates provided 4
Certificate Revocation List http://crl.comodoca.com/COMODORSADomainValidationSecureServerCA.crl
OCSP URI http://ocsp.comodoca.com
OCSP stapling offered
OCSP must staple no
DNS CAA RR (experimental) not offered
Certificate Transparency no
Testing HTTP header response @ "/"
HTTP Status Code 200 OK
HTTP clock skew +7 sec from localtime
Strict Transport Security 182 days=15768000 s, just this domain
Public Key Pinning --
Server banner nginx
Application banner --
Cookie(s) (none issued at "/")
Security headers --
Reverse Proxy banner --
Testing vulnerabilities
Heartbleed (CVE-2014-0160) not vulnerable (OK), timed out
CCS (CVE-2014-0224) not vulnerable (OK)
Ticketbleed (CVE-2016-9244), experiment. not vulnerable (OK)
Secure Renegotiation (CVE-2009-3555) not vulnerable (OK)
Secure Client-Initiated Renegotiation not vulnerable (OK)
CRIME, TLS (CVE-2012-4929) not vulnerable (OK)
BREACH (CVE-2013-3587) potentially NOT ok, uses gzip HTTP compression. - only supplied "/" tested
Can be ignored for static pages or if no secrets in the page
POODLE, SSL (CVE-2014-3566) not vulnerable (OK)
TLS_FALLBACK_SCSV (RFC 7507) Downgrade attack prevention supported (OK)
SWEET32 (CVE-2016-2183, CVE-2016-6329) VULNERABLE, uses 64 bit block ciphers
FREAK (CVE-2015-0204) not vulnerable (OK)
DROWN (CVE-2016-0800, CVE-2016-0703) not vulnerable on this host and port (OK)
make sure you don't use this certificate elsewhere with SSLv2 enabled services
https://censys.io/ipv4?q=F5E30EA7EA319419848E1F024566135244E223B50147118CE02A635D345672F8 could help you to find out
LOGJAM (CVE-2015-4000), experimental not vulnerable (OK): no DH EXPORT ciphers, no common primes detected
BEAST (CVE-2011-3389) TLS1: ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA
AES128-SHA AES256-SHA DHE-RSA-CAMELLIA256-SHA CAMELLIA256-SHA
DHE-RSA-CAMELLIA128-SHA CAMELLIA128-SHA DES-CBC3-SHA
VULNERABLE -- but also supports higher protocols (possible mitigation): TLSv1.1 TLSv1.2
LUCKY13 (CVE-2013-0169), experimental potentially VULNERABLE, uses cipher block chaining (CBC) ciphers with TLS
RC4 (CVE-2013-2566, CVE-2015-2808) no RC4 ciphers detected (OK)
Testing 364 ciphers via OpenSSL plus sockets against the server, ordered by encryption strength
Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (RFC)
-----------------------------------------------------------------------------------------------------------------------------
xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
x9f DHE-RSA-AES256-GCM-SHA384 DH 2048 AESGCM 256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
x6b DHE-RSA-AES256-SHA256 DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
x39 DHE-RSA-AES256-SHA DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
x88 DHE-RSA-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384
x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256
x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA
x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
xc027 ECDHE-RSA-AES128-SHA256 ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
x9e DHE-RSA-AES128-GCM-SHA256 DH 2048 AESGCM 128 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
x67 DHE-RSA-AES128-SHA256 DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
x33 DHE-RSA-AES128-SHA DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
x45 DHE-RSA-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256
x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256
x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA
x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
x0a DES-CBC3-SHA RSA 3DES 168 TLS_RSA_WITH_3DES_EDE_CBC_SHA
Running client simulations via sockets
Android 2.3.7 TLSv1.0 DHE-RSA-AES128-SHA, 2048 bit DH
Android 4.1.1 TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
Android 4.3 TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
Android 4.4.2 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 5.0.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 6.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Android 7.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Chrome 51 Win 7 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Chrome 57 Win 7 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Firefox 49 Win 7 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Firefox 53 Win 7 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
IE 6 XP No connection
IE 7 Vista TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
IE 8 XP TLSv1.0 DES-CBC3-SHA
IE 8 Win 7 TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
IE 11 Win 7 TLSv1.2 DHE-RSA-AES128-GCM-SHA256, 2048 bit DH
IE 11 Win 8.1 TLSv1.2 DHE-RSA-AES128-GCM-SHA256, 2048 bit DH
IE 11 Win Phone 8.1 Update TLSv1.2 DHE-RSA-AES128-GCM-SHA256, 2048 bit DH
IE 11 Win 10 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Edge 13 Win 10 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Edge 13 Win Phone 10 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Opera 17 Win 7 TLSv1.2 ECDHE-RSA-AES128-SHA256, 256 bit ECDH (P-256)
Safari 5.1.9 OS X 10.6.8 TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
Safari 7 iOS 7.1 TLSv1.2 ECDHE-RSA-AES128-SHA256, 256 bit ECDH (P-256)
Safari 9 OS X 10.11 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Safari 10 OS X 10.12 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Apple ATS 9 iOS 9 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Tor 17.0.9 Win 7 TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
Java 6u45 No connection
Java 7u25 TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
Java 8u31 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
OpenSSL 1.0.1l TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
OpenSSL 1.0.2e TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
Done 2018-02-15 21:58:43 [ 307s] -->> 69.89.75.196:443 (www.techempower.com) <<--
I tentatively propose TLS versions of two existing test types: Plaintext and Fortunes. Tentatively, TLS-Plaintext and TLS-Fortunes.
The TLS-Plaintext will hardly test the handshake and mostly test the encoding costs.
However, the encoding costs might depend more on the SSL/TLS library (OpenSSL / BearSSL etc') rather than the framework (I'm ignoring buffer consolidation at the moment, although this is definitely a consideration where pipelining is concerned).
I'm wondering if the performance testing costs are worth the possible data mining that would be enabled by the TLS-Plaintext test...?
including a TLS test type is reasonable as long as there are some stacks for which TLS would be included "in-stack." And I know that is satisfied in the real-world because many stacks include nginx or another mainstream web server that has robust TLS capability.
Sidenote: Technically speaking, nginx (and friends) has a known issue with forward security and session cache invalidation. I'm not sure if resolving these issues would effect performance, but I'm quite certain that security is more important than any performance performance penalty that patching these issues might incur.
I'm pointing this out because this might be something we want to test (I have no idea how to do that, though).
Keep in mind that this is a framework benchmark. Choosing to use openssl/bearssl is a choice of the framework of language authors. I don't see why this is any less of a framework test because of that choice.
Absolutely, and from my digging in frameworks the way they use openssl etc can make a massive difference, including unneeded memory copies and locks. What I hope is that this benchmark will highlight and put pressure on these things to be changed.
If every framework ends up with the same Delta cost for https overtime then this test would have achieved it's goal in my mind.
Another 2 minor points:
- The client and the database machine at ServerCentral did not have
AES-NIinstructions support. This should be taken into account for the cipher choice(s) or checked not to be limiting factor; - Some data on the client and the server should be gathered with
openssl speed- with and without the-evpoption and the chosen cyphers.
I'm using this one: https://testssl.sh/, https://github.com/drwetter/testssl.sh I think it will tick every box: mainly shell script, non-exotic external dependencies, similar functionality to SSL Labs tests.
@zloster That looks fantastic. We'll probably also want to ensure that the load generator uses the agreed-to cipher suite (whatever that ends up being) and only that cipher suite.
Sidenote: Technically speaking, nginx (and friends) has a known issue with forward security and session cache invalidation. I'm not sure if resolving these issues would effect performance, but I'm quite certain that security is more important than any performance performance penalty that patching these issues might incur.
@boazsegev I agree that to whatever degree it is possible, I would like to test configurations that are as secure as current best-practices suggest. If/when there is a resolution to the issue you cite, it would be good to apply that to the test implementations.
Keep in mind that this is a framework benchmark. Choosing to use openssl/bearssl is a choice of the framework of language authors. I don't see why this is any less of a framework test because of that choice.
@RX14 is correct on this point. If the framework has specified a preferred (canonical) TLS provider, we'd want to use that in these test implementations. As we have done elsewhere, in cases where a preferred provider is not specified, we'd like to use a mainstream option such as OpenSSL.
What I hope is that this benchmark will highlight and put pressure on these things to be changed.
@Drawaes Agreed! This is my sentiment generally across everything we do in this project. :)
The client and the database machine at ServerCentral did not have AES-NI instructions support. This should be taken into account for the cipher choice(s) or checked not to be limiting factor;
@zloster We are bringing a new environment (named "Citrine" in the results dashboard for the time being), and these servers include AES-NI.
The point raised regarding testing TLS handshake performance (aka for new connections) vs. encoding/encryption costs is an interesting one, and makes me wonder whether we should configure the Fortunes variant to not allow connection re-use. Or perhaps we need a separate test altogether, e.g. TLS-Handshake, that is effectively plaintext without connection re-use (which of course implies no pipelining too). Having a different test might help avoid any conflation of traditional Fortunes results with the TLS variant.
I, for one, certainly think being able to see a framework's performance characteristics for TLS handshakes vs. encryption is valuable.
One point you will potentially be measuring a whole stack of areas that just aren't measured today and might tell you very little about TLS performance per se.
If this is likely to be a test that is looked at first I would recommend there is a test with zero connection reuse without TLS ( unless I am mistaken and this is what Fortunes is today?) As you would want a baseline for establishing the socket etc before moving onto the handshake itself.
Reason being is that if you have an equal field configuration without session resumption then the only difference in reality is the extra round trips of tiny messages. From my measurements the actual cryptographic calculations are a minimal part of the handshake.
Then of course you have to decide do you allow session resumption/ticketing and which keyshares/curves etc which massively expands the problem.
@Drawaes
If this is likely to be a test that is looked at first I would recommend there is a test with zero connection reuse without TLS ( unless I am mistaken and this is what Fortunes is today?)
https://en.wikipedia.org/wiki/HTTP_persistent_connection#HTTP_1.1 Connection reuse AKA "keep-alive" should be the default behaviour for all tests. The plaintext is using additionally "request pipelining" i.e. issuing new GET operations without receiving the response from the previous HTTP operation (https://en.wikipedia.org/wiki/HTTP_pipelining).
Which is what I thought ;) I guess my concern people will look at this test and think... Wow TLS is expensive when in the real world keep alive is the norm and there is no unencrypted baseline to compare to.
I believe the JSON test would be a better choice than the plaintext test when testing TLS.
As I mentioned here, the plaintext pipelining approach exposes servers to DoS attacks through clients that pipeline large quantities of short requests (preferably requesting a large payload).
A 48 byte request can be repeated ~11 times in a single TCP/IP packet when MTU=576 (which is the common limitation on the web).
An attacker could make hundreds of requests, combined with a slow client approach, to starve a server of resources (for example, consider memory consumption for the buffered responses).
There are two defense measures a security aware framework might implement: 1. throttling (slowing down pipelined requests); and 2. limiting (limiting the number of incomplete downloads for slow clients).
This means that security aware frameworks are likely to perform slower on plaintext tests due to pipelining.
This would result in an unfair advantage for frameworks that ignore this possible attack vector.
Hence, I believe that the JSON test would provide a better representation where security is concerned, since it avoids this question altogether.
Just my 2¢... also, I'm somewhat biased (facil.io throttles pipelined requests).
Hi folks, I'm a bit of a noob regarding all this, but I have some feedback.
I just ran a crude same-machine test for aspcore/json manually and put a https endpoint in. I got roughly a third rps less with ssl (47K vs 70K), which is substantial and was not on my radar when I saw the impressive rps values of the benchmarks (or the concerns raised by @boazsegev).
I don't know if connections were kept alive during my test, the keep-alive header had no effect on the result. I assume the expensive part of ssl is the handshake and done only once for each connection. Is it possible to tell wrk to re-open each connection? There's no obvious option.
