Self Signed Certs preventing Login

[Jaffythethird]

Summary I'm on a company VPN, trying to make a leapp integration for a customer environment. Using the browser, I can sign in (it requires having https in the URL, but it returns as "insecure" because of the self signed cert) through the AWS SSO. However, using the Leapp integration causes nothing to pop up, even when browser is selected. Using the CLI, I can now see why that is. Error code is written below.

Version Leapp Version 0.13.4

Expected Response: I understand why insecure connections may be blocked, but it seems to fail without displaying an error to the user (unless using the CLI), and it would be nice to have an --insecure override if there's not other way of resolving this. How might I get this to connect for now?

CLI Response (leapp integration login):

node:events:491
    throw er; Unhandled 'error' event
    ^
{
Error: self signed certificate in certificate chain
    at TLSSocket.onConnectSecure (node:_tls_wrap:1534:34)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:events:489:12)
    at TLSSocket._finishInit (node:_tls_wrap:948:8)
    at ssl.onhandshakedone (node:_tls_wrap:729:12)
Emitted 'error' event on Writable instance at:
    at eventHandlers.<computed> (/usr/local/lib/node_modules/@noovolari/leapp-cli/node_modules/follow-redirects/index.js:14:24)
    at ClientRequest.emit (node:events:513:28)
    at ClientRequest.emit (node:events:489:12)
    at TLSSocket.socketErrorListener (node:_http_client:481:9)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
  code: 'SELF_SIGNED_CERT_IN_CHAIN'
}

Node.js v18.7.0

Desktop:

• OS: MacOS Monterey
• OS Version 12.5
• Leapp Version 0.13.4

[ghost]

Good catch!

What happens if you add the self-signed cert or the CA it was created with into the Keychain as trusted CA?

[urz9999]

Good catch indeed! The problem is that this error was not watched correctly, so there is no proper error handling for it at the moment, nonetheless will get a look into it, as we are planning another release soon

[Jaffythethird]

What happens if you add the self-signed cert or the CA it was created with into the Keychain as trusted CA?

Sorry for the late response, but I've been trying things like that. None of them worked for me, though I am admittedly not the most experienced at Keychain/Cert management. Nonetheless, I'm pretty sure that the several ways I went about adding them into the Keychain as trusted included a correct way of doing it. No fix, though.

The problem is that this error was not watched correctly, so there is no proper error handling for it at the moment, nonetheless will get a look into it, as we are planning another release soon

Do you think that the error handling will allow working around this problem, or just making sure that the user is notified as to why they can't get their login to start?

[ghost]

Sorry for the late response, but I've been trying things like that. None of them worked for me, though I am admittedly not the most experienced at Keychain/Cert management. Nonetheless, I'm pretty sure that the several ways I went about adding them into the Keychain as trusted included a correct way of doing it. No fix, though.

Thank you for testing. I can try to replicate the issue and work this out by creating similar AWS SSO environment unless @urz9999 is already on it?

Do you think that the error handling will allow working around this problem, or just making sure that the user is notified as to why they can't get their login to start?

A bit of both, depending how this issue will get fixed. I was thinking instructing the user to for example turning "allow insecure connection" (not yet implemented) setting on or assist with marking certificate as trusted. It should work also as a warning for possible man-in-the-middle attack if user suddenly starts seeing the error with previously working connection.

[urz9999]

Actually, you are free to try it by yourself currently, I'm on a new feature that hopefully will be released next week

[ghost]

It seems that Node uses it's own CA store and that's why fiddling with Key Chain didn't work out. I'll add possibility to import own CA/cert, option to connect without certificate checking and add exception handling.

[tgjohnst]

Chiming in to say I'm testing a VPN+firewall solution with SSL inspection and am also affected by this issue.

AWSCLI had a similar issue because it uses the certs vendored by botocore, rather than trusting the system keychain. That was fixed by either appending the custom certs we use to the botocore bundle or pointing AWSCLI to a custom bundle via either the ca_bundle var in .aws/config or the AWS_CA_BUNDLE environment variable.

The ability to provide a custom cert bundle (we have at least 2 certs that need trusting), add trusted certs to whichever bundle Leapp is using, disable SSL validation, or at the very least a place where Leapp's cert bundle lives that's editable would be extremely helpful. Is there currently a timeline for implementation of this, or a feature branch open? Or any workaround that wouldn't involve building Leapp from source?

[ghost]

Thank you for chiming in.

I have a working fix for this and are working on dialogs to detect and import required certs. I can probably in few days create a pr for a version with basic import functionality and if it goes smoothly, also with a checkbox to disable ssl validation for the session. It's not going to be a global option, at least yet.

By two different certs, do you mean a cert-chain with CA and intermediate certs or just a cert-bundle with two different certs and if latter, do you need to have both certs for one session or one cert for few sessions and other for another sessions?

I mean is it enough for now if you can assign one cert for a specific IAM User or IAM Federated-session or AWS SSO integration or do you have a need for two different trusted certs to trust for one session or SSO integration?

Maybe I should also implement an application wide setting for using proxy.

But first things first. I'll create a PR for ability to import cert or cert-bundle for a session.

[tgjohnst]

@c-sami Thank you, and great questions. I'm relatively new to cert management and working on grokking all that's involved here but am happy to test/review/assist with the PR whenever it's ready!

We have an organization-wide firewall/VPN doing SSL inspection with it's own root cert. By two different certs, I mean a cert chain with CA + intermediate, and these would be needed for all sessions when on VPN, they're not session-specific in my context. The ability to have a trusted cert [bundle] for a session or disable SSL validation for a session would certainly be a valid solution.

[ghost]

Hi,

I built an UNOFFICIAL build for testing skipping certificate validation. Version has UI for adding custom ca certificates, but it is disabled as the implementation is work-in-progress.

https://github.com/siipimutteri/siipimutteri-leapp/releases (My own UNOFFICIAL build. Not related to Noovolari. Use at your own risk! Will remove soon. ONLY FOR TESTING THIS ISSUE) Built from: https://github.com/siipimutteri/siipimutteri-leapp/tree/1a4d4f7a4d64920f2cade404166171d6b1d0009e Build job: https://github.com/siipimutteri/siipimutteri-leapp/actions/runs/3029033658

[ghost]

This is the proposal for a new "Network"-tab that can be accessed from the upper-right "options"-button. Depending how much there is space, I could also move proxy-settings under it. Final version does not have so much text and the warning-disclaimer will be accessible under tooltip.

But for this testing build, I wanted to leave the disclaimer clearly visible.

[andreacavagna01]

Let me know if that build solves the problem. At this time, I'm unable to test it, then we will work to add a solution in the app in the next releases

[ghost]

I tested it just by creating own CA, intermediate CA and server cert, firing up nginx docker container with created server certificate and pointed AWS SSO integration into it. Without the skip-option, I got SSL error and with it login error.

And that reminded me that I forgot to cover IAM-users! Maybe I'll be able to push a new test release even today. And use 0.14.2 as a base. Now it was based on 0.14.1.

At least for Windows, it is possible to skip installation, unpack the zip archives and just start the Leapp from the downloaded zip.

[tgjohnst]

Wow, thank you for the quick turnaround!

I downloaded the test build and was able to test. ~~Unfortunately was unable to test on my machine as the network tab was incomplete and the user defined CA field was unresponsive/uninteractible when clicked. Not sure what would cause this. One guess - I'm running on a Macbook Pro M1, is the test build for macOS targeted at ARM64?~~

edit - missed your note about the custom cert box being disabled. Although my certificate validation checkbox was missing the warning text in your screenshot, can confirm that the test build does successfully restore function on VPN (where we have MITM happening) with the skip validation option enabled! I did notice the build was a lot slower in general to start and interact with, but guessing that's not due to this change in particular.

Tested on MacOS 12.5.1, ARM64 (M1), with an AWS SSO session

[ghost]

Sorry about the delay. I'm implementing this with a best-effort basis and are currently resolving issues with all other connections in addition to sso sessions. It turned out that there are multiple places that needs same certificates and maybe system certificates could be used. It needs at least three different implementations. One for each supported OS. Testing and validating has been a bit problematic even with using mitmproxy.

[ghost]

Still working on this. Slowly though because lack of time. Experimenting if win-ca, mac-ca and linux-ca or system-ca would work or is own implementation the only way.

[tgjohnst]

Thank you for the updates, @c-sami

[siipimutteri]

c-sami "ghost" here. I just changed from my work account to personal one. I have been fiddling around with this using transparent mitmproxy and I think I got it finally working by using 'system-ca'. Only troubles are with windows-build (damned node-gyp) and cli. I tested mac-ca that hung the whole app, built my own cert-sync-module with rust and lots of different things. But anyhow.. now we are getting close.

~~Here's current setup: https://github.com/siipimutteri/siipimutteri-leapp/commit/277325aab8692866cd1084dbfd0b7413ab0df2dc Just ignore .github workflow file and concentrate on system-ca and systemCertsAsync lines.~~ edit: Here's clean version: https://github.com/siipimutteri/siipimutteri-leapp/commit/4ac16ee23e96e40e74867a8d00276a29257a74fe Funny thing is that at one point I had hundreds lines of code and now only few lines that are actually what seemed to be needed.

I'll publish linux and mac (intel & arm) binaries for testing. Windows build will be available after I have figured out what's the issue with rebuilding node-gyp.

[siipimutteri]

Unofficial builds for testing: https://github.com/siipimutteri/siipimutteri-leapp/releases/tag/0.16.2-126cf39.1-320 Cli not working. ~~No Windows build yet.~~

I have tested ~~only Mac Arm64 build~~ mac x64 and arm64 builds.

To run in mac, you'll need to bypass security warning with ctrl+click as I don't have Apple Developer subscription and didn't sign the binary.

[siipimutteri]

Tested with Mac x64 and it works. Noticed that we don't have win32 build, but not sure if we need one.

Testing with Linux and Windows is still todo. Catching the cert error is still an issue. Leapp integration login stops silently working if there is a certificate error and Leapp needs to be restarted.

[siipimutteri]

Windows 64bit build is also available. Node-gyp issue is strange, but got around it by using npm ci instead of npm install

[tgjohnst]

Can confirm that build Leapp-0.16.2-126cf39.1-mac-x64 works and successfully gets around the certificate issue while logged in on VPN on my M1 macbook running macOS Ventura. I don't have a Windows or Linux machine on our VPN so am unable to test in either of those environments. Any other updates to this effort?

[ericvilla]

Firstly, thank you @siipimutteri for the commitment! Given @tgjohnst feedback, the only OS that's still not tested is Linux. I think we can merge the proposed Pull Request into the main branch, test it, and release it during the next week.

[siipimutteri]

There are still few issues about handling SSL and connection error events that I have been trying to debug and sort out. For being able to release the long awaited feature, I'll create an opt-in setting for enabling "Trust system CA (beta)". I think I can do the PR for it this weekend.

Currently GUI works but CLI doesn't, when using mitmproxy and trusting mitmproxy's CA. It might be a mitmproxy issue or a wider one. Maybe @tgjohnst can test the CLI after the release.

[ericvilla]

Can confirm that build Leapp-0.16.2-126cf39.1-mac-x64 works and successfully gets around the certificate issue while logged in on VPN on my M1 macbook running macOS Ventura. I don't have a Windows or Linux machine on our VPN so am unable to test in either of those environments. Any other updates to this effort?

@tgjohnst once logged into AWS SSO, are you able to start Leapp AWS SSO Role Sessions via VPN?

[siipimutteri]

Figured out and fixed the issue with the CLI. Now AWS SSO Integration login works with desktop app and cli. I replaced wireguard&mitmproxy with pfSense Plus, OpenVPN and Squid to get more authentic environment. Next step is to fully test AWS SSO integration and then implement feature with other access methods, integrations and update notifier.

[ericvilla]

@Jaffythethird we're currently working with Sami to introduce this feature in the next release.

I'm wondering what are your configuration details: is the customer AWS environment limiting access to AWS SSO APIs to a specific range of sourceIp addresses? In this case, I'm quite sure it applies to all AWS SDK API calls. Is the VPN server deployed in the customer environment?

[Jaffythethird]

Admittedly, I've gotten into a workflow that doesn't require leapp. That being said, I appreciate all the work being put into this.

And, as for the customer environment... I've not been with them for a while and can't say for sure what the configuration was. I will say, since it deals with compliance stuff, it was probably delegated to FIPS endpoints

[tgjohnst]

Can confirm that build Leapp-0.16.2-126cf39.1-mac-x64 works and successfully gets around the certificate issue while logged in on VPN on my M1 macbook running macOS Ventura. I don't have a Windows or Linux machine on our VPN so am unable to test in either of those environments. Any other updates to this effort?

@tgjohnst once logged into AWS SSO, are you able to start Leapp AWS SSO Role Sessions via VPN?

sorry for the late reply, missed the ping. Yes, to answer in further detail:

• realized that the last time, I was testing with the x64 build even on an arm64 system. I tried the arm64 version and macOS reported that it's broken, so something is wrong with the arm64 build for this release (Leapp-0.16.2-126cf39.1-mac-arm64). Proceeded testing with -x64
• In the main release branch, while on VPN, the process of logging into an SSO integration and starting an SSO session would hang indefinitely without popping up a browser window. Can confirm that starting Leapp build Leapp-0.16.2-126cf39.1-mac-x64, connecting to the VPN, and logging into an AWS SSO integration + starting a session works correctly on the x64 build via GUI. Screenshot attached

Any updates on the developer side?

[siipimutteri]

@ericvilla has inspected the issue and found out that fixing the issue for all session types needs a lot of work. I haven’t done any action for a while, but now that you pinged, I see a need for the fix and maybe we could introduce the existing fix for the sso integration only and fix other session types later? I believe fix for sso sessions is almost ready for release.

I can start working with fixing other session types as I have now all supported session types at my hand, including federation via GoogleWorkspaces (GApps).