Building cl-http2-protocol in 2021

[Symbolics]

I got the system to build, but it won't run and thought it would be useful to document how to get this far so that if someone else wants to pick up the ball and run with it they'll have a head start.

First, you'll want to review the contents of issue #4. Those are just symptoms of the fact that cl-async and cl+ssl both wrap the OpenSSL libs (why doesn't cl-async use cl+ssl?). @martinflack describes at a high level what remains to be done to finish off the project. I don't think any of this is insurmountable, but it will take a bit of time to organise all the pieces.

The immediate fixes to get the system building in this repo are:

• Add :cl-libevent2-ssl to the list of dependencies in the ASDF file
• Replace use with mix in the example package definition
• Rename the second argument in the generic function emit

These changes, and a few others, are in the Symbolics fork. A pull request has been made to bring these into this repo.

The remaining issues are in the dependent libraries, mostly cl-async and its dependencies, which seems to have suffered some minor bit rot.

1. If on MS Windows, the SSL libs won't load because CFFI can't locate the libraries. The fix has been submitted as a pull request orthecreedence/cl-async#187. If this patch hasn't made it into the main branch, you'll need to make the change yourself.
2. cl-async in turn has a dependency on cl-libevent2, a CFFI wrapper over libevent. The wrapper code was generated with swig, which has removed support for Common Lisp in its latest version. cl-libevent2 uses an older version of libevent. There are a few options:
   • downgrade swig and regenerate the bindings with a newer libevent
   • downgrade libevent
   • do nothing and hope that the current bindings work with the new libevent
   • try generating the bindings with the CFFI groveler

Whilst this may get cl-async compiling on a developers workstation, it's not going to be a very nice user experience for others who want to use the library.

Now you can build cl-http2-protocol. The build will contain errors because of references to internal cl-async functions, now removed. See #4 for an explanation of why. The build errors aren't that great in number. Someone needs to look at each of the internal cl-async functions called and check for a public interface equivalent. Ideally do this working with the upstream maintainer to get any changes adopted there. It would probably be good for both projects.

There's some other tidying up that could be done in cl+ssl, but there don't appear to be any blockers there.

; file: s:/src/http2/tcp.lisp
; in: CL-ASYNC-UTIL:DEFINE-C-CALLBACK CL-ASYNC::TCP-EVENT-CB
;     (CL-ASYNC::GET-LAST-TCP-ERR)
; 
; caught STYLE-WARNING:
;   undefined function: CL-ASYNC::GET-LAST-TCP-ERR


; file: s:/src/http2/example.lisp
; in: DEFUN CL-HTTP2-PROTOCOL-EXAMPLE:EXAMPLE-CLIENT
;     (CL-ASYNC-SSL::INIT-SSL-CLIENT-CONTEXT CL+SSL::*SSL-GLOBAL-CONTEXT*)
; 
; caught STYLE-WARNING:
;   undefined function: CL-ASYNC-SSL::INIT-SSL-CLIENT-CONTEXT


; file: s:/src/http2/tcp-ssl.lisp
; in: DEFUN CL-ASYNC-SSL:TCP-SSL-SERVER
;     (CL-ASYNC-SSL::LAST-SSL-ERROR)
; 
; caught STYLE-WARNING:
;   undefined function: CL-ASYNC-SSL::LAST-SSL-ERROR


; file: s:/src/http2/tcp.lisp
; in: CL-ASYNC-UTIL:DEFINE-C-CALLBACK CL-ASYNC::TCP-EVENT-CB
;     (CL-ASYNC::RELEASE-DNS-BASE)
; 
; caught STYLE-WARNING:
;   undefined function: CL-ASYNC::RELEASE-DNS-BASE


; file: s:/src/http2/tcp-ssl.lisp
; in: DEFUN CL-ASYNC-SSL:TCP-SSL-SERVER
;     (CL-ASYNC-SSL::TCP-SERVER-C CL-ASYNC-SSL::SERVER)
; 
; caught STYLE-WARNING:
;   undefined function: CL-ASYNC-SSL::TCP-SERVER-C

;     (CL-ASYNC-SSL::TCP-SERVER-DATA-POINTER CL-ASYNC-SSL::SERVER)
; 
; caught STYLE-WARNING:
;   undefined function: CL-ASYNC-SSL::TCP-SERVER-DATA-POINTER


; file: s:/src/http2/tcp.lisp
; in: CL-ASYNC-UTIL:DEFINE-C-CALLBACK CL-ASYNC::TCP-EVENT-CB
;     (CL-ASYNC-UTIL:RUN-EVENT-CB CL-ASYNC::EVENT-CB CL-ASYNC:EVENT)
; --> HANDLER-BIND SB-KERNEL::%HANDLER-BIND LET SB-INT:DX-FLET FLET 
; --> #:H0 BLOCK SETF 
; ==>
;   (SETQ CL-ASYNC::_EVCB-ERR CL-ASYNC::E)
; 
; caught WARNING:
;   undefined variable: CL-ASYNC::_EVCB-ERR
; 
; compilation unit finished
;   Undefined functions:
;     CL-ASYNC::GET-LAST-TCP-ERR CL-ASYNC-SSL::INIT-SSL-CLIENT-CONTEXT CL-ASYNC-SSL::LAST-SSL-ERROR CL-ASYNC::RELEASE-DNS-BASE CL-ASYNC-SSL::TCP-SERVER-C CL-ASYNC-SSL::TCP-SERVER-DATA-POINTER
;   Undefined variable:
;     CL-ASYNC::_EVCB-ERR
;   caught 1 WARNING condition
;   caught 19 STYLE-WARNING conditions

[zellerin]

Would it make sense - in 202x - to get rid of some issues with move from NPN to ALPN? It appears to me that the ssl.lisp could go away completely, as at least SNI is supported now, and ALPN seems to be easy to fix in cl+ssl (at least client side, ~30 lines of code)..