tls: handshake failure

[pana67]

Hi, I am trying to make work enrolment lets encrypt certificate for our ASA firewall / VPN gateway. I am able to communicate with ASA REST API, I installed I hope all required software on internal linux server (Debian),but when I run the command for enrolment I obtain error:

• The following errors were reported by the server:

  Domain: fw.networksys.cz Type: tls Detail: remote error: tls: handshake failure

  To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address. Additionally, please check that you have an up-to-date TLS configuration that allows the server to communicate with the Certbot client.

---

it seems to me the certificate is not published on ASA but I do not know how to debug it - I enclose the whole CLI output for details certbot-asa-log.txt . On the ASA there is already certificate from our internal (private) CA - I do not know how the certificate from certbot-asa script is setup on the ASA, as there is no attribute for the name of output interface.

Thank you for any help.

[chrismarget]

I've seen this error myself recently.

The issue was that when LE's authenticator connected to the ASA to validate the challenge certificate, it wasn't able to establish a TLS session with the ASA.

I didn't figure out what was wrong, but suspect it's an SSL cipher support issue on the ASA. At the time I had the problem, the same LE authenticator (same IP address, anyway) was able to talk fine with one of my ASAs, but not with the other.

There was no time to debug the problem at that time, so I didn't pursue it further.

What version of software is running on the ASA? I'll see if I can replicate the problem.

[pana67]

Hi Chris,

I am running the last available ASA-OS - 9.8(1) ( I need at least 9.7.1 as there is VTI interface configured) I am quite skilled in ASA so I can try to debug SSL connection on the ASA side – that does not apply for my skills in linux :-o

Regards

Pavel

From: Chris Marget [mailto:[email protected]] Sent: Friday, June 30, 2017 5:14 PM To: chrismarget/certbot-asa [email protected] Cc: Navrátil Pavel [email protected]; Author [email protected] Subject: Re: [chrismarget/certbot-asa] tls: handshake failure (#1)

I've seen this error myself recently.

The issue was that when LE's authenticator connected to the ASA to validate the challenge certificate, it wasn't able to establish a TLS session with the ASA.

I didn't figure out what was wrong, but suspect it's an SSL cipher support issue on the ASA. At the time I had the problem, the same LE authenticator (same IP address, anyway) was able to talk fine with one of my ASAs, but not with the other.

There was no time to debug the problem at that time, so I didn't pursue it further.

What version of software is running on the ASA? I'll see if I can replicate the problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/chrismarget/certbot-asa/issues/1#issuecomment-312294636, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcauW_okzholqS3NVL5wINDH3DQd5BxMks5sJRDEgaJpZM4OKqpe.

[chrismarget]

So, if the failure is happening where I think it's happening...

If you run 'show run | inc trustpoint|trust-point' repeatedly on the ASA while the plugin is operating, you should see a temporary trustpoint and SNI configuration appear briefly.

Once those are installed, we're just waiting for the LE authenticator to come look at them.

At this point, you should be able to see the TLS failure if you're watching packets (fairly easy to spot), or with the right debug incantation on the ASA (I'm not sure what that might be).

I'll fire up an ASA in the lab.

[pana67]

Hi,

you are right – I can see the temporary acme challenge trustpoint for a while.

fw# sh runn | in acme crypto ca trustpoint acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 keypair acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 crypto ca certificate chain acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 ssl trust-point acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 domain 85bd3442b8ece3d01183ae375610bea5.6b7ac57b435eac0a38c306f42d8b61bd.acme.invalid

I tried to set ssl debug – I can see error

error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher@s3_srvr.c:2053

in time the server should check the certificate

I hope this can help

Regards

Pavel

From: Chris Marget [mailto:[email protected]] Sent: Friday, June 30, 2017 5:54 PM To: chrismarget/certbot-asa [email protected] Cc: Navrátil Pavel [email protected]; Author [email protected] Subject: Re: [chrismarget/certbot-asa] tls: handshake failure (#1)

So, if the failure is happening where I think it's happening...

If you run 'show run | inc trustpoint|trust-point' repeatedly on the ASA while the plugin is operating, you should see a temporary trustpoint and SNI configuration appear briefly.

Once those are installed, we're just waiting for the LE authenticator to come look at them.

At this point, you should be able to see the TLS failure if you're watching packets (fairly easy to spot), or with the right debug incantation on the ASA (I'm not sure what that might be).

I'll fire up an ASA in the lab.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/chrismarget/certbot-asa/issues/1#issuecomment-312304968, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcauW7BRLKGi7dShxWedc9-KN_0JzoDfks5sJRorgaJpZM4OKqpe.

[pana67]

Hi,

I found it will probably be problem in cypher set - the error message "no shared cipher" should mean there is no supported encryption cipher on ASA - I tried to allow all kinds of cypher on ASA but it ends with the same error.

Regards

Pavel

From: Navrátil Pavel Sent: Friday, June 30, 2017 7:21 PM To: 'chrismarget/certbot-asa' [email protected]; chrismarget/certbot-asa [email protected] Cc: Author [email protected] Subject: RE: [chrismarget/certbot-asa] tls: handshake failure (#1)

Hi,

you are right – I can see the temporary acme challenge trustpoint for a while.

fw# sh runn | in acme crypto ca trustpoint acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 keypair acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 crypto ca certificate chain acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 ssl trust-point acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 domain 85bd3442b8ece3d01183ae375610bea5.6b7ac57b435eac0a38c306f42d8b61bd.acme.invalid

I tried to set ssl debug – I can see error

error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher@s3_srvr.c:2053mailto:cipher@s3_srvr.c:2053

in time the server should check the certificate

I hope this can help

Regards

Pavel

From: Chris Marget [mailto:[email protected]] Sent: Friday, June 30, 2017 5:54 PM To: chrismarget/certbot-asa <[email protected]mailto:[email protected]> Cc: Navrátil Pavel <[email protected]mailto:[email protected]>; Author <[email protected]mailto:[email protected]> Subject: Re: [chrismarget/certbot-asa] tls: handshake failure (#1)

So, if the failure is happening where I think it's happening...

If you run 'show run | inc trustpoint|trust-point' repeatedly on the ASA while the plugin is operating, you should see a temporary trustpoint and SNI configuration appear briefly.

Once those are installed, we're just waiting for the LE authenticator to come look at them.

At this point, you should be able to see the TLS failure if you're watching packets (fairly easy to spot), or with the right debug incantation on the ASA (I'm not sure what that might be).

I'll fire up an ASA in the lab.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/chrismarget/certbot-asa/issues/1#issuecomment-312304968, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcauW7BRLKGi7dShxWedc9-KN_0JzoDfks5sJRorgaJpZM4OKqpe.

[pana67]

And one more probably useful info - these are the set supported by default on ASA:

fw# sh ssl ciphers Current cipher configuration: default (medium): ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-GCM-SHA384 AES256-GCM-SHA384 ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES256-SHA384 DHE-RSA-AES256-SHA256 AES256-SHA256 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256 AES128-GCM-SHA256 ECDHE-ECDSA-AES128-SHA256 ECDHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA256 AES128-SHA256 DHE-RSA-AES256-SHA AES256-SHA DHE-RSA-AES128-SHA AES128-SHA tlsv1 (medium): DHE-RSA-AES256-SHA AES256-SHA DHE-RSA-AES128-SHA AES128-SHA tlsv1.1 (medium): DHE-RSA-AES256-SHA AES256-SHA DHE-RSA-AES128-SHA AES128-SHA tlsv1.2 (medium): ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-GCM-SHA384 AES256-GCM-SHA384 ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES256-SHA384 DHE-RSA-AES256-SHA256 AES256-SHA256 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256 AES128-GCM-SHA256 ECDHE-ECDSA-AES128-SHA256 ECDHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA256 AES128-SHA256 DHE-RSA-AES256-SHA AES256-SHA DHE-RSA-AES128-SHA AES128-SHA dtlsv1 (medium): DHE-RSA-AES256-SHA AES256-SHA DHE-RSA-AES128-SHA AES128-SHA

Regards

Pavel

From: Navrátil Pavel Sent: Friday, June 30, 2017 7:36 PM To: 'chrismarget/certbot-asa' [email protected]; 'chrismarget/certbot-asa' [email protected] Cc: 'Author' [email protected] Subject: RE: [chrismarget/certbot-asa] tls: handshake failure (#1)

Hi,

I found it will probably be problem in cypher set - the error message "no shared cipher" should mean there is no supported encryption cipher on ASA - I tried to allow all kinds of cypher on ASA but it ends with the same error.

Regards

Pavel

From: Navrátil Pavel Sent: Friday, June 30, 2017 7:21 PM To: 'chrismarget/certbot-asa' [email protected]; chrismarget/certbot-asa <[email protected]mailto:[email protected]> Cc: Author <[email protected]mailto:[email protected]> Subject: RE: [chrismarget/certbot-asa] tls: handshake failure (#1)

Hi,

you are right – I can see the temporary acme challenge trustpoint for a while.

fw# sh runn | in acme crypto ca trustpoint acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 keypair acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 crypto ca certificate chain acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 ssl trust-point acme_challenge_0ce0e19e6b60fd64350fbc9277aad043 domain 85bd3442b8ece3d01183ae375610bea5.6b7ac57b435eac0a38c306f42d8b61bd.acme.invalid

I tried to set ssl debug – I can see error

error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher@s3_srvr.c:2053mailto:cipher@s3_srvr.c:2053

in time the server should check the certificate

I hope this can help

Regards

Pavel

From: Chris Marget [mailto:[email protected]] Sent: Friday, June 30, 2017 5:54 PM To: chrismarget/certbot-asa <[email protected]mailto:[email protected]> Cc: Navrátil Pavel <[email protected]mailto:[email protected]>; Author <[email protected]mailto:[email protected]> Subject: Re: [chrismarget/certbot-asa] tls: handshake failure (#1)

So, if the failure is happening where I think it's happening...

If you run 'show run | inc trustpoint|trust-point' repeatedly on the ASA while the plugin is operating, you should see a temporary trustpoint and SNI configuration appear briefly.

Once those are installed, we're just waiting for the LE authenticator to come look at them.

At this point, you should be able to see the TLS failure if you're watching packets (fairly easy to spot), or with the right debug incantation on the ASA (I'm not sure what that might be).

I'll fire up an ASA in the lab.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/chrismarget/certbot-asa/issues/1#issuecomment-312304968, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcauW7BRLKGi7dShxWedc9-KN_0JzoDfks5sJRorgaJpZM4OKqpe.

[chrismarget]

Yup, this is where I wound up last time it happened.

It's not really an issue with the plugin, but the TLS service on the ASA rejecting LE's TLS client.

I couldn't figure it out last time, but I intend to take another crack at it.

[chrismarget]

Pavel, can you test something for me?

You'll have to be quick: During the interval when the ASA is configured to serve up the challenge certificate, but before LE gets around to checking... Please try to delete the ssl trust-point acme-challenge_blah domain blah line from the ASA configuration?

My hope is that the error will change from something about TLS handshaking to a challenge validation failure (indicating that TLS succeeded).

I'm finding that with that directive in place I have trouble with the TLS handshake on my test box.

[chrismarget]

I think we're running into this Cisco bug. An ASA upgrade is going to be required.

[pana67]

Hi Chris,

I just post a request for help on Cisco Support Forum - will see if somebody answer to it. https://supportforums.cisco.com/discussion/13325616/ssl-handshake-failure

I sniffed the SSL communication - enclosing the file with it - you can check it weather you will be able to get some useful info from it.

Regards

Pavel

From: Chris Marget [mailto:[email protected]] Sent: Saturday, July 1, 2017 12:57 PM To: chrismarget/certbot-asa [email protected] Cc: Navrátil Pavel [email protected]; Author [email protected] Subject: Re: [chrismarget/certbot-asa] tls: handshake failure (#1)

I think we're running into thishttps://bst.cloudapps.cisco.com/bugsearch/bug/CSCve20346 Cisco bug. An ASA upgrade is going to be required.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/chrismarget/certbot-asa/issues/1#issuecomment-312425434, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcauW6uOFzLAM4Pe-b-SvIY7gxjPX28oks5sJiX3gaJpZM4OKqpe.

[chrismarget]

I'm confident that the bug I mentioned is your problem. There's a couple of options for verifying that:

1. Downgrade to 9.3(2). The problem should go away.

2. Manually:

• Hit CTRL-C a couple of times as soon as certbot says INFO:certbot.auth_handler:Waiting for verification... Don't let certbot clean up after itself. The ASA will be left with a trustpoint, RSA keypair and an ssl trust-point directive that weren't there before. You can remove these later.
• Look at the ssl trust-point line. The last argument is a very long domain name ending in .acme.invalid
• Knock against the ASA using openssl: openssl s_client -connect <ASA-IP-Address>:443 -servername <long-domain-name.acme.invalid>

The s_client command should fail with SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure. That's the bug.

Remove the ssl trust-point line, or change the -servername argument to some other value and the s_client command should succeed (meaning you don't get your Linux shell prompt back). At that point, type GET / HTTP/1.0 <CR><CR> and the ASA should spit some HTML at you (through the TLS session created by openssl).

It seems that any TLS connection which matches the SNI filter specified by the ssl trust-point <blah> domain <blah> directive will have a problem because of this bug. Frankly, I'm surprised you got past the openssl s_client step in the plugin's readme, because it relies on SNI.