quicklisp-client
quicklisp-client copied to clipboard
quicklisp
Use HTTPS instead of HTTP
Hi!
I've noticed that archives are downloaded via http:
CL-USER> (ql:quickload :thread-pool)
To load "thread-pool":
Load 2 ASDF systems:
arnesi bordeaux-threads
Install 1 Quicklisp release:
thread-pool
Downloading http://beta.quicklisp.org/archive/thread-pool/2012-01-07/thread-pool-20120107-git.tgz
##########################################################################
But I found that they are also available if I change schema to https:
[art@art-osx5:~]% curl -v -s https://beta.quicklisp.org/archive/thread-pool/2012-01-07/thread-pool-20120107-git.tgz > /dev/null
* Trying 52.85.242.94...
* Connected to beta.quicklisp.org (52.85.242.94) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.quicklisp.org
* Server certificate: Gandi Standard SSL CA 2
* Server certificate: USERTrust RSA Certification Authority
* Server certificate: AddTrust External CA Root
> GET /archive/thread-pool/2012-01-07/thread-pool-20120107-git.tgz HTTP/1.1
> Host: beta.quicklisp.org
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 3061
< Connection: keep-alive
< Date: Sun, 03 Jun 2018 19:14:56 GMT
< Last-Modified: Sat, 07 Jan 2012 21:52:07 GMT
< ETag: "9dfcb3dd5692d474d90f7916722d5bf8"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 450
< X-Cache: Hit from cloudfront
< Via: 1.1 f9a0ddc3860252ab6c4d02ab024b4891.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: _WSj_KutsQ1kyoFACC3wQs3zq8jbyo5QjQXIcrCb1DJi4mCZKz2CVw==
<
{ [3061 bytes data]
* Connection #0 to host beta.quicklisp.org left intact
It would be great (and more secure) to switch to the HTTPS. What do you think, @xach?
Here is some system info:
CL-USER> (cl-info:make-cl-info)
OS: Darwin 15.6.0
Lisp: SBCL 1.4.3
ASDF: 3.3.1.1
QL: org.borodust.bodge 20180214223017
quicklisp 2017-10-23
CL-USER> (ql:update-client)
Downloading http://beta.quicklisp.org/client/quicklisp.sexp
##########################################################################
The most up-to-date client, version 2017-03-06, is already installed.
T
CL-USER> (ql:client-version)
"2017-03-06"
It would be good to do, but there's no straightforward path to do it. Implementations do not all provide HTTPS support, it's not straightforward to make it from scratch or use HTTPS libraries on all supported platforms.
I hope to soon roll out a different form of authentication of downloads via signatures.
Alternatively, for the time being I figure it is possible to point quicklisp at a http proxy that will convert requests to https, I think?
SHA256 verification of downloads or similar might be easier to implement than HTTPS?...
(Of course, both would be better eventually.)
SHA256 verification of downloads or similar might be easier to implement than HTTPS?...
(Of course, both would be better eventually.)
I don't know if I'm just stupid but I think SHA256 verification doesn't help without HTTPS. I think you need some kind of signature (like pgp) as @quicklisp already suggested
You communicate the SHA256 hashes in advance in a secure way authenticated by a signature (as part of the dist metadata download), and then later you download over HTTP (or HTTPS) and verify that the hashes of the files are the expected ones...
@Hexstream So yeah you meant a combination. I thought you meant to only use SHA256 sums.
By the way, guys, have you seen an alternative package manager https://gitlab.common-lisp.net/clpm/clpm which is able to use a quicklisp distribution and to force HTTPS?
I've found it recently but didn't have time to try.
I guess integrity is more important than privacy in our case, so the common method would use something like a GPG key to sign the list of package checksums upon distro build. Then, once downloaded and verified, that list can be used to verify individual packages.
That would mean another external dependency and calling inferiors through OS. GPG binaries are readily available on most platforms though.
Otherwise one could depend on ironclad for PK cryptography and signing/verification of the package cksum list. One of GPG's "strengths" is that the distribution signing key could be acquired/verified trough already existing key server infrastructure.
I have Quicklisp client code for sha256 checking and GPG signature checking in portable CL. See the gpg branch for more info.
Right now the hold up is key management and a bit of infrastructure updates.
I note in passing that Keybase is a pretty great modern alternative to the fairly antiquated GPG infrastructure.
One straightforward approach to handling HTTPS is graceful degradation. You already have HTTP connections with GPG signature checking as the fall-through. Adding platform-specific support is unpleasant, but it's either here or in a dedicated portable SSL library that will have to be bundled into Quicklisp.
From there, make it highly visible when using HTTPS. When not, keep the same interface we're used to.
Is that sufficient or am I overlooking anything?
Is this still in progress? I created cl-micropm for some of Quicklisp's limitations (CLPM development seems to have stopped), but it would be nice to have support for HTTPS and Github/Gitlab sources. Or if there's a way to move out Quicklisp's dependency resolution algorithm so that we can build stuff on top of it -- I tried a couple of times to track it down and pull it out in the code but with no success.
One alternative to implementing a sufficient amount of GPG in CL is to port OpenBSD's signify instead. It's a lot simpler in comparison and they rely on Ed25519 which is already implemented in CL as part of Ironclad.
The gpg stuff is already complete (see the "pgp" branch). It's a matter of working out key policies and updating dist building and dist fetching at this point.
It's not that easy to reuse parts of ironclad.
On Fri, Oct 7, 2022 at 1:15 PM Vasilij Schneidermann < @.***> wrote:
One alternative to implementing a sufficient amount of GPG in CL is to port OpenBSD's signify instead. It's a lot simpler in comparison and they rely on Ed25519 which is already implemented in CL as part of Ironclad.
— Reply to this email directly, view it on GitHub https://github.com/quicklisp/quicklisp-client/issues/167#issuecomment-1271848169, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACPNLPSNM5GNJDUH3CYV5LWCBLCVANCNFSM4FDA2FKA . You are receiving this because you were mentioned.Message ID: @.***>
The pgp branch is from 4 years ago, of course it would be great having proper key policies and signature verification, but in the spirit of not letting perfect be the enemy of good, what about incorporating some changes from https://github.com/rudolfochrist/ql-https in the meantime? It would be:
- Shell out to curl for https downloads. curl is now present on almost all unix, mac, and windows boxes, in other words 99% of users. If curl is not available for https download, then download as it's currently doing over http but show a warning
- Check file hash and size from release dist for the downloads. This is currently md5sum but if quicklisp dist could be published with sha256sum instead it would be a lot better. Again can just shell out to md5sum or sha256sum and print a warning when it's not available, but on 99% of users it is available. This makes it so if an attacker can get a https cert for quicklisp.org, or temporarily compromise the quicklisp servers and backdoor some releases, they'd have to maintain the compromise for months until users download a new dist containing the hash and size for the attacker's backdoored versions of packages. Of course if these dists are signed with PGP even better.
But in summary just shelling out to curl for https download if it's available, and printing a warning otherwise, would be a simple and immediate and significant improvement in quicklisp security. I think it's quite worthwhile to implement in the meantime as it seems dist signing may be some more years off.
Shameless plug (again), but I wrote a small, single file package manager for CL (<200 LOC) that reads the Quicklisp sources but prefers git for decentralization, and creates project-local dependencies.
The solution may be to include implementation-specific parts of CFFI (cffi-sys package) for minimal FFI, and use WinHTTP on Windows, OpenSSL library on Linux, and similar operating systems(OpenSSL is virtually everywhere, and version 3.0 includes HTTP(S) client, and older versions are out of support anyway), and otherwise default to plain HTTP over sockets(on Mezzano?).
Not sure about the OpenSSL library situation on macOS though, but that OS has something called Core Foundation and its C API includes http-related stuff.
Any updates on this?
@bo-tato would you be open to submitting a fork with your suggestion? I think it's a great idea since it takes care of most of the problem.
@quicklisp would you accept a fork?
@quicklisp What do you think about the following:
How many systems are there that don't support https anyway?
I'm sure at this stage it's probably easier to make github/gitlab repositories for those than have to deal with the special cases for the security issues. You don't have to do this though. It should be the responsibility of those systems' maintainers.
It's not so unreasonable to require https in 2024 and only passing a (:allow-non-encrypted-sources T) to be able to download systems that do not provide encrypting. Default should be encrypted and I don't think it's up to quicklisp to have to worry about how to deal with those systems. It's 2024 already...
If you want to be backwards compatible to not upset the community, you could then invert it and have it (:only-encrypted-sources T) instead, though I think that's just simply not up to basic security standards and should really be unnecessary. To the point that if you insist on keeping backwards compatibility it almost makes sense to have a new interface that defaults to encryption and have it be the standard way quicklisp works, and just keep the old way to avoid breaking backwards compatibility. It seems like it's only really one function so you can just name it (ql:load)...
In conclusion, this change should be rather simple and straightforward, with the burden placed on the system maintainers, it can even keep backwards compatibility, and probably 99.9% of users will benefit tremendously.
Any thoughts?
@quicklisp Is there any issue bigger than this for quicklisp?
If you are too busy, can you at least comment on what type of implementation are you willing to accept and open it up to the community to submit a pull request for this?
Btw, thanks for all the work on quicklisp in general! 😄
@bo-tato would you be open to submitting a fork with your suggestion? I think it's a great idea since it takes care of most of the problem.
https://github.com/rudolfochrist/ql-https is the fork that implements those suggestions (well just checking md5 not sha256 as we can't change the hash format of main quicklisp dist). If quicklisp wants to incorporate those changes I can work on a PR that better merges it into quicklisp proper.
I recently noticed Quicklisp was downloading over HTTP. If @quicklisp would accept it I was thinking about making a PR that would merge the changes from ql-https into Quicklisp. I think this is a reasonable solution given that the user probably will have already installed curl, since the instructions at https://www.quicklisp.org/beta/ tell them to.
Alternatively, if there's some way I can help with the pgp branch, I could do that. I'm not too familiar with how that stuff works, but if there's any way I can help with this issue I'd be happy to.
