71 Protocol error on AWS lambda function

[zueve]

I have service hosted on AWS: API Gateway + Lambda function technologies. Curl return correct response:

curl -vvv https://2fp1vl8r05.execute-api.ap-southeast-1.amazonaws.com/dev/api/accounts/4489/test/?a=1&b=5

* Connected to 2fp1vl8r05.execute-api.ap-southeast-1.amazonaws.com (13.32.226.49) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.execute-api.ap-southeast-1.amazonaws.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=*.execute-api.ap-southeast-1.amazonaws.com
*        start date: Wed, 06 Dec 2017 00:00:00 GMT
*        expire date: Thu, 06 Dec 2018 12:00:00 GMT
*        issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /dev/api/accounts/4489/test/?a=1 HTTP/1.1
> Host: 2fp1vl8r05.execute-api.ap-southeast-1.amazonaws.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Content-Length: 81
< Connection: keep-alive
< Date: Thu, 04 Oct 2018 06:52:15 GMT
< x-amzn-RequestId: 075f1015-c7a2-11e8-ba31-e1edd8f7551f
< Allow: GET, HEAD, OPTIONS
< x-amzn-Remapped-Content-Length: 81
< x-amz-apigw-id: OOnI9HQpyQ0FhDQ=
< x-amzn-Remapped-WWW-Authenticate: HTTP_X_SSL_DN realm="api"
< X-Amzn-Trace-Id: Root=1-5bb5b89f-3696619802c92882113f3d2c;Sampled=0
< X-Cache: Error from cloudfront
< Via: 1.1 6dc745d2e515f93f6ccc56a06bdbb8dc.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: CaW8WqV1bLwlNfyoHhV6a7_35R3Bp9PwIPD40QFXHP2lp-SrBH4Pyw==
<
* Connection #0 to host 2fp1vl8r05.execute-api.ap-southeast-1.amazonaws.com left intact
{"message":"Incorrect authentication credentials.","code":"AuthenticationFailed"}
[1]+  Done                    curl -vvv https://2fp1vl8r05.execute-api.ap-southeast-1.amazonaws.com/dev/api/accounts/4489/test/?a=1

But yandex-tank wrote only 2 +1 100.00% : 71 Protocol error

load.yaml:

phantom:
  ssl: true
  address: 2fp1vl8r05.execute-api.ap-southeast-1.amazonaws.com:443
  header_http: "1.1"
  headers:
    - "[Host: 2fp1vl8r05.execute-api.ap-southeast-1.amazonaws.com]"
    - "[User-Agent: curl/7.47.0]"
    - "[Accept: */*]"
  uris:
    - /dev/api/accounts/4489/test/?a=1&b=5
  load_profile:
    load_type: rps
    schedule: line(1, 5, 5s)
console:
  enabled: true
telegraf:
  enabled: false

It's looks like same as #176 but server return correct status line.

In phantom*.log I found:

2018-10-04 07:12:03.782 +0000 [info] [] Start
2018-10-04 07:12:03.787 +0000 [error] [benchmark_io 016] format error
2018-10-04 07:12:03.933 +0000 [error] [benchmark_io 000] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:03.934 +0000 [error] [benchmark_io 000] SSL_connect error 1
2018-10-04 07:12:04.750 +0000 [error] [benchmark_io 002] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:04.750 +0000 [error] [benchmark_io 002] SSL_connect error 1
2018-10-04 07:12:05.275 +0000 [error] [benchmark_io 001] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:05.275 +0000 [error] [benchmark_io 001] SSL_connect error 1
2018-10-04 07:12:05.701 +0000 [error] [benchmark_io 004] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:05.702 +0000 [error] [benchmark_io 004] SSL_connect error 1
2018-10-04 07:12:06.092 +0000 [error] [benchmark_io 006] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:06.093 +0000 [error] [benchmark_io 006] SSL_connect error 1
2018-10-04 07:12:06.459 +0000 [error] [benchmark_io 008] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:06.463 +0000 [error] [benchmark_io 008] SSL_connect error 1
2018-10-04 07:12:06.786 +0000 [error] [benchmark_io 010] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:06.786 +0000 [error] [benchmark_io 010] SSL_connect error 1
2018-10-04 07:12:07.038 +0000 [error] [benchmark_io 012] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:07.039 +0000 [error] [benchmark_io 012] SSL_connect error 1
2018-10-04 07:12:07.403 +0000 [error] [benchmark_io 003] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:07.403 +0000 [error] [benchmark_io 003] SSL_connect error 1
2018-10-04 07:12:07.693 +0000 [error] [benchmark_io 005] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:07.694 +0000 [error] [benchmark_io 005] SSL_connect error 1
2018-10-04 07:12:07.848 +0000 [error] [benchmark_io 007] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:07.849 +0000 [error] [benchmark_io 007] SSL_connect error 1
2018-10-04 07:12:08.074 +0000 [error] [benchmark_io 009] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:08.075 +0000 [error] [benchmark_io 009] SSL_connect error 1
2018-10-04 07:12:08.360 +0000 [error] [benchmark_io 011] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:08.360 +0000 [error] [benchmark_io 011] SSL_connect error 1
2018-10-04 07:12:08.518 +0000 [error] [benchmark_io 013] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:08.518 +0000 [error] [benchmark_io 013] SSL_connect error 1
2018-10-04 07:12:08.761 +0000 [error] [benchmark_io 014] SSL error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
2018-10-04 07:12:08.762 +0000 [error] [benchmark_io 014] SSL_connect error 1
2018-10-04 07:12:08.764 +0000 [error] [monitor_io] bq_sleep: Operation canceled
2018-10-04 07:12:08.764 +0000 [error] [phantom_logger] bq_sleep: Operation canceled
2018-10-04 07:12:08.765 +0000 [error] [monitor_io] bq_sleep: Operation canceled
2018-10-04 07:12:08.765 +0000 [error] [benchmark_io stream_method brief_logger] bq_sleep: Operation canceled
2018-10-04 07:12:08.767 +0000 [info] [] Exit

It's strange because other lambda functions works fine and I use official Docker image for run yandex-tank

[zueve]

I found reason for different behaviour on different was lambda functions. It depends on configure Endpoint parameter in API-Gatewey: if Endpoint=Region (default) - all works fine if Endpoint=Egde - SSL_connect error

Endpoint=Egde connect with AWS CloudFront (CDN) proxy

[shirkevich]

Getting same as @zueve 71 Protocol error when using tank against host behind CloudFlare with TLS v1.0-1.1 allowed.

[afilatov]

Getting same as @zueve 71 Protocol error when using tank against host behind CloudFlare with TLS v1.0-1.1 allowed.

This could happen probably due to the lack of proper TLS SNI support in yandex-tank with phantom generator. If so, I believe you have several options to mitigate this issue with yandex-tank:

• Use different load generator within yandex-tank: pandora, jmeter...
• Patch this function in phantom according to this gist. This will work only if you have single hostname that doesn't change frequently as you will need to recompile phantom;

[gorodnov]

Hi @shirkevich were you able to resolve your issue with 71 Protocol error ?

[Izerrion]

After some extensive testing, we've found out that this 71 error in our case was related to purely configured ammo files and not phantom issues inside a TCP stack. Weirdly enough, tank launches with these ammo files successfully, but always yields 71 protocol errors.

We made some changes to the ammo file and we managed to bypass that issue. For example, we had:

<request_size#1> test GET <our_api_URL> HTTP/1.1 Host: <our_api_Endpoint> User-Agent: tank Accept: application/json, text/plain, */* Special_header: <special_header_value#1> Special_header: <special_header_value#2> Connection: close

Then, we added a required invisible symbol "^M" (carriage return) like this: (keep in mind that request size changed because of the carriage return)

<request_size#2> test GET <our_api_URL> HTTP/1.1 ^M Host: <our_api_Endpoint> ^M User-Agent: tank ^M Accept: application/json, text/plain, */* ^M Special_header: <special_header_value#1> ^M Special_header: <special_header_value#2> ^M Connection: close ^M ^M

After that, tank started sending requests and they yielded successful result instead of old 71 error. This issue is more apparent if you are using your own custom scripts to generate the ammo instead of the built-in "ammo.py" for different headers load inside the same ammo file. Hope it helps someone.

P.S. You can use VIM as your text editor to show these hidden characters by default. https://unix.stackexchange.com/questions/32001/what-is-m-and-how-do-i-get-rid-of-it