[FEATURE] - Support HTTPS with Insecure / Self-signed Certificates

[briandowns]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your feature request related to a problem?

I may have missed an option but it doesn't seem as though there's the ability to use SSL with self signed certificates.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[Jason2605]

I'm not entirely sure I understand what you mean sorry? Could you provide a little more context, thanks!

[briandowns]

Yeah, my fault. I was in a rush to get the issue opened and didn't expand on it. Apologies. I was writing an HTTP client service for Opnsense. And now looking through to get the error, I tweaked one stupid thing and it was fine. What it initially looked like was that it was refusing to talk to the server since it had a self signed certificate. No actual issue.

[Jason2605]

No problems at all, glad you sorted it!

[briandowns]

Looks like this is an issue when speaking to an HTTPS endpoint with a self signed certificate. I receive the following error:

SSL peer certificate or SSH remote key was not OK

If I add the following lines in the get function implementation, the call goes through successfully.

curl_easy_setopt(curl, CURLOPT_SSL_VERIFYSTATUS, 0);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);

I was thinking of adding another optional arg to the get and post functions to indicate if the call should ignore "bad" certificates. Was hoping to get your thoughts on the matter. @Jason2605

[Jason2605]

Ah okay, thats an interesting problem. My thinking with those options to cURL you're essentially turning off the verification completely kind of defeating the point of the certificate, I think it may be better if we add the ability to pass a path to a cert file while making the HTTP request.

A quick google leads me to this docs page: https://curl.se/libcurl/c/CURLOPT_CAINFO.html but unfortunately I haven't yet tested it myself to know for sure this is the option we'd want

[briandowns]

I'd say there are situations where we'd want to be able to communicate with systems that are running self signed certifcates. For example, I use this a lot for testing of various things.

In Go you have the ability to tell the HTTP client to ignore bad certificates like this:

    tr := &http.Transport{
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
    }
    client := &http.Client{Transport: tr}
    _, err := client.Get("https://golang.org/")
    if err != nil {
        fmt.Println(err)
    }

Having something this simple would be great.

[Jason2605]

Sorry I don't think I was super clear, the above would allow you to handle self-signed certs it's just that you would need to provide the path to your cert as it's not an "accepted" one by default, rather than disabling the checks in their entirety.

Although, there are probably cases where we want both scenarios so it would be interesting how we handle that without making the function signatures for the HTTP module unwieldily.

Maybe something along the lines of passing a "config" object (like in your Go example) as the last parameter that define these sorts of options that aren't used massively, but definitely could come up, what do you think?

[briandowns]

I like the idea of a config object very much. Seems like it'll keep the function signatures tidy while still allowing for expansion and potential change.

Not to harp on Go too much but it has the concept of a client that can be reused and eventually gc'd when done. Have we reached a point where we do something similar? Of course, keep the existing get and post calls but also add an HTTP client that can be configurable for reuse?

[Jason2605]

Don't worry about the language comparisons, if anything I welcome it! I think the HTTP library definitely needs a rethink, it's rather rudimental in it's approach (only exposing 2 methods, response being rather simple, no configurable client like you mentioned), all of these are things that will need to change at some point.

Will be interesting to compare approaches across popular implementations and see what we can do to streamline the module

[briandowns]

Toying around with a few ideas and came up with this and was curious for your feedback, @Jason2605.

import HTTP;
import JSON;
import System;

// main
{
    const opts = {
        "timeout": 20,
        "headers": [
            "Content-Type: application/json", 
            "Accept: application/json",
            "User-Agent: Dictu"
        ],
        "insecure": true,
        "key_file": "",
        "cert_file": ""
    }
    const ret = HTTP.newClient(opts);
    if (not ret.sucess()) {
        print(res.unwrapError());
        System.exit(1);
    }
    var httpClient = ret.unwrap();

    var res = httpClient.get("https://localhost:4433");
    if (not res.success()) {
        print(res.unwrapError());
        System.exit(1);
    }
    print(res.unwrap().statusCode);
}

[Jason2605]

I like it, very clean. Will be much better with the client object too like you proposed!

[briandowns]

I think we can close this with the addition of the client opts.

[Jason2605]

Resolved as of #548