Added new OpenSSL IO Handler for OpenSSL 1.1.1

[mezen]

Complete new io handler which supports OpenSSL 1.1.1. The existing io handler for OpenSSL 1.0.2 is untouched and should still work. Only IdCTypes.pas and IdGlobal.pas are modified (the last files in the diff): I added some more types.

The Binaries from http://wiki.overbyte.eu/wiki/index.php/ICS_Download#Download_OpenSSL_Binaries_.28required_for_SSL-enabled_components.29 are used, but at the beginning I tested with slWebPro.

The complete io handler is written with Delphi Berlin 10.1.2, I tried to support older Delphi versions, but due lack of Installations I havent tested it. Also only Win32 and Win64 are tested, no MacOS, no mobile. Test are done in small test project and a big (three tier) real world application with tcp server/client, smtp/imap/pop client, ftp client and http client.

This should fix #213, #224 (ok, no event, but virtual method for overriding), #248 (not on purpose for this issue, but for myself - found out later that issue exists^^) and of course #183.

[rlebeau]

Just an FYI, when TIdFTP is preparing a new data connection, it Clone()'s the SSLIOHandler assigned to the command channel. When TIdSSLIOHandlerSocketOpenSSL is cloned, it does reuse the same session ID as the original SSLIOHandler, but currently only when connecting an outbound connection (ie, for an FTP passive-mode transfer), not when accepting an inbound connection (ie, for an FTP active-mode transfer). So make sure that your new 1.1.x SSLIOHandler handles session ID reuse when cloned, if it doesn't already.

[mezen]

My last commit ensures that with the Clone() call, the session is always passed to the clone, so the later connection direction is irrelevant.

[JedrzejczykRobert]

Modified source for OpenSSL 1.1.1

OpenSSL.zip

Compile and tested with:

Delphi 7 + Win XP sp2 Delphi 2007 + Win XP sp3 Delphi XE 7 + Win 7 Delphi XE 10.4 + Win 10, Linux64, MacOS-10.15 Lazarus 2.0.10 + Win10, Win 7, WinXP, Ubuntu-18, MacOS-10.15

[xjikka]

Very impressive work. It is working by me on XE6, Delphi 10.2, 10.3, 10.4. To avoid update Indy and avoid changes in Indy IdGlobal.pas and Idctypes.pas, I've added new IdOpenSSL_IdCTypesEx.pas with missing consts and types. To make it working in Linux (tested on Raspbian) in Lazarus, I had to edit / replace SafeLoadLibrary with HackLoad in IdOpenSSLLoader.pas

  {$IFDEF MSWINDOWS}
  LLibCrypto := SafeLoadLibrary(FOpenSSLPath + CLibCrypto);
  LLibSSL := SafeLoadLibrary(FOpenSSLPath + CLibSSL);
  {$ELSE}
  LLibCrypto := HMODULE(HackLoad(FOpenSSLPath + CLibCryptoLinuxRaw,SSLDLLVers));
  LLibSSL := HMODULE(HackLoad(FOpenSSLPath + CLibSSLLinuxRaw,SSLDLLVers));
  {$ENDIF} 

and add constants to IdOpenSSLConsts.pas

CLibCryptoLinuxRaw = 'libcrypto'; CLibSSLLinuxRaw = 'libssl'; SSLDLLVers : array [0..1] of string = ('','.1.1');

[rlebeau]

To avoid update Indy and avoid changes in Indy IdGlobal.pas and Idctypes.pas, I've added new IdOpenSSL_IdCTypesEx.pas with missing consts and types.

Why would you want to avoid that, if that is where they belong? What exactly did you add that needed to be put into its own unit?

[xjikka]

To avoid update Indy and avoid changes in Indy IdGlobal.pas and Idctypes.pas, I've added new IdOpenSSL_IdCTypesEx.pas with missing consts and types.

Why would you want to avoid that, if that is where they belong? What exactly did you add that needed to be put into its own unit?

Normally I would do it as You do, but I can't upgrade / change Indy source in all versions of Delphi I have. Because of that, I made a separate IdOpenSSL_IdCTypesEx.pas in which the missing types and procedures are added, and therefore I don't need to change and recompile Embardadero\Studio\xx.0\source\Indy10. I have also added IdOpenSSL_IdCTypesEx.pas to "uses" in units, where needed. Finally, to implement support of new TLS13 into existing instalation of XE6, 10.2, 10.3 and Lazarus I must only add path to patched OpenSSL source directory in delphi-options-language-delphi-librarypath and nothing more.

IdOpenSSL_IdCTypesEx.zip Updated 18.8.2020

[SlMaker]

Any ETA for the official merge to master?

[xjikka]

Any ETA for the official merge to master?

I'm sorry, I have no experience with Git at all, but if You're interested, I can provide You changed source (zipped).

[grahamegrieve]

I needed to make some changes to the code for FPC (v3.3.1):

• %line% creates a string constant, so %LINE% -> %LINENUM% IdOpenSSLHeaders_crypto
• parameters can't be called "out" - IdOpenSSLHeaders_cmac and IdOpenSSLHeaders_conf
• System.Types --> Types in IdOpenSSLHeaders_x509v3 and IdOpenSSLHeaders_crypto

I should update this PR?

[grahamegrieve]

And on Linux:

• IdOpenSSLHeaders_ssl case of x509 is wrong
• IndyCheckWindowsVersion in IdOpenSSLIOHandlerClientBase needs to be $IFDEF MSWINDOWS

Also, there's a number of getProcAddress that fail on the standard lib

[grahamegrieve]

Failed to load list below. These are pretty long lists that will generate lots of confusion. Should we $IFDEF these out?

Windows + Linux (ubuntu 20): BIO_s_datagram_sctp BIO_new_dgram_sctp BIO_dgram_is_sctp BIO_dgram_sctp_wait_for_dry BIO_dgram_sctp_msg_waiting BIO_f_zlib _CONF_new_section _CONF_get_section _CONF_add_string _CONF_get_string _CONF_get_number _CONF_new_data _CONF_free_data CRYPTO_mem_debug_push CRYPTO_mem_debug_pop CRYPTO_get_alloc_counts CRYPTO_mem_debug_malloc CRYPTO_mem_debug_realloc CRYPTO_mem_debug_free CRYPTO_mem_leaks_cb CRYPTO_mem_leaks ebcdic2ascii ascii2ebcdic BIO_set_md EVP_rc5_32_12_16_cbc EVP_rc5_32_12_16_ecb EVP_rc5_32_12_16_cfb64 EVP_rc5_32_12_16_ofb RAND_query_egd_bytes RAND_egd RAND_egd_bytes SSL_CTX_set_tlsext_use_srtp SSL_set_tlsext_use_srtp SSL_get_selected_srtp_profile SSL_trace

Windows only: EC_GFp_nistp224_method EC_GFp_nistp256_method EC_GFp_nistp521_method

linux only: EVP_idea_ecb EVP_idea_cfb64 EVP_idea_ofb EVP_idea_cbc IDEA_options IDEA_ecb_encrypt IDEA_set_encrypt_key IDEA_set_decrypt_key IDEA_cbc_encrypt IDEA_cfb64_encrypt IDEA_ofb64_encrypt IDEA_encrypt

[grahamegrieve]

@mezen This doesn't appear to work as a server for TIdHTTPServer - any connection generates

'error:140940F4:SSL routines:ssl3_read_bytes:unexpected message'.

The server set up code is

FServer := TIdHTTPServer.Create(Nil);
FServer.Scheduler := TIdSchedulerOfThreadPool.Create(nil);
FServer.DefaultPort := FActualSSLPort;
FServer.KeepAlive := SECURE_KEEP_ALIVE;
FIOHandler := TIdOpenSSLIOHandlerServer.Create(Nil);
FServer.IOHandler := FIOHandler;

FIOHandler.Options.CertFile := FCertFile;
FIOHandler.Options.CertKey := FKeyFile;
FIOHandler.Options.VerifyCertificate := FRootCertFile;
FIOHandler.Options.OnGetPassword := SSLPassword;

It doesn't matter what options I set. I think that that SSL negotiation is working ok - if I set to verify the client certificate that part works ok

[SlMaker]

Failed to load list below. These are pretty long lists that will generate lots of confusion. Should we $IFDEF these out?

Windows + Linux (ubuntu 20): _CONF_new_section _CONF_get_section _CONF_add_string _CONF_get_string _CONF_get_number _CONF_new_data _CONF_free_data CRYPTO_mem_debug_push CRYPTO_mem_debug_pop CRYPTO_get_alloc_counts CRYPTO_mem_debug_malloc CRYPTO_mem_debug_realloc CRYPTO_mem_debug_free CRYPTO_mem_leaks_cb CRYPTO_mem_leaks

1. _CONF could be removed, sounds like it's nothing you can ever use in code
2. The debug ones are probably only available if you've compiled OpenSSL with debug option.

Windows only: EC_GFp_nistp224_method EC_GFp_nistp256_method EC_GFp_nistp521_method

OpenSSL documentation: "Note, however, that these implementations are not available on all platforms."

Think it's not a major problem as all of them are not needed by Indy code thus the programmer should check if it was loaded before using it in own code.

[grahamegrieve]

Agree that it's not a major problem but the problem is that routine practice in the old code is to check FailedToLoad and raise an exception if it's not empty. That fact that this list is not empty normally will generate a stream of questions in all the forums

[grahamegrieve]

@mezen dsa_st and rsa_st are defined in both IdOpenSSLHeaders_ossl_typ and IdOpenSSLHeaders_evp and the definitions of EVP_PKEY_get0_RSA etc appear wrong - should be prsa not prsa_st etc

[grahamegrieve]

It appears that the definitions of EVP_PKEY_set1_RSA and EVP_PKEY_set1_DSA are wrong:

from openSSL doco:

int EVP_PKEY_set1_RSA(EVP_PKEY *pkey, RSA *key); int EVP_PKEY_set1_DSA(EVP_PKEY *pkey, DSA *key);

proposed pascal code:

EVP_PKEY_set1_RSA: function(pkey: PEVP_PKEY): TIdC_INT cdecl = nil; EVP_PKEY_set1_DSA: function(pkey: PEVP_PKEY): TIdC_INT cdecl = nil;

[grahamegrieve]

Also this code should be in the loader (see attachment) indy-fips.txt

(uses IdGlobal, IdCTypes, IdFIPS, IdOpenSSLHeaders_ossl_typ, IdOpenSSLHeaders_rsa, IdOpenSSLHeaders_dsa, IdOpenSSLHeaders_bn, IdOpenSSLHeaders_bio, IdOpenSSLHeaders_hmac, IdOpenSSLHeaders_pem, IdOpenSSLHeaders_err, IdOpenSSLHeaders_x509, IdOpenSSLHeaders_evp, IdOpenSSLHeaders_crypto, IdOpenSSLVersion)

[grahamegrieve]

And the types are wrong here:

EVP_DigestSignInit: function(ctx: PEVP_MD_CTX; pctx: PPEVP_PKEY_CTX; const &type: PEVP_MD; e: PENGINE; pkey: PEVP_PKEY): TIdC_INT cdecl = nil; EVP_DigestSignFinal: function(ctx: PEVP_MD_CTX; sigret: PByte; siglen: PIdC_SIZET): TIdC_INT cdecl = nil;

[mezen]

Ok, now some feedback came in, so one after the other:

@JedrzejczykRobert I have implemented most of your changes

@xjikka also adopted

@grahamegrieve here it gets a bit longer

• "%LINE% -> %LINENUM%" adopted
• parameters can't be called "out" -> already noted and adopted by @JedrzejczykRobert with his changes. In fact you can already call it out if you prefix it with &. What I didn't know, that it didn't work in older Delphi versions
• "System.Types -> Types" also already included in @JedrzejczykRobert
• "IdOpenSSLHeaders_ssl case of x509 is wrong" you can enter a line number, I can't find what you mean
• "IndyCheckWindowsVersion in IdOpenSSLIOHandlerClientBase" adopted
• Not successful loaded functions: As my previous posters have already written, it depends a lot on the conditions your binary was built with - unless @rlebeau explicitly wants it to be different, I would rather leave it that way, since it is not a bug and works. The list FailedToLoad is rather interesting if you want to use OpenSSL outside of the IO handler
• "dsa_st and rsa_st are defined in both IdOpenSSLHeaders_ossl_typ and IdOpenSSLHeaders_evp" yep, corrected
• "EVP_PKEY_set1_RSA wrong" not after the correct type is used. In the set you cannot simply omit what is set in the 'EVP_PKEY'!
• "EVP_DigestSignInit & EVP_DigestSignFinal are wrong" I have looked at https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/include/openssl/evp.h#L634 again and the definition looks so correct, can you describe the error in more detail?

[grahamegrieve]

• "IdOpenSSLHeaders_ssl case of x509 is wrong" line 59 - text is "IdOpenSSLHeaders_X509", must be "IdOpenSSLHeaders_x509" (matters for FPC on OSX)
• Not successfully loaded: we should at a minimum document that this is not expected to be empty. It would be useful to add a list of functions that are nil for the standard distributions, maybe. That might depend on how many problems it causes
• EVP_PKEY_set1_RSA wrong: the doco for openSSL:

int EVP_PKEY_set1_RSA(EVP_PKEY *pkey, RSA *key);

your header:

EVP_PKEY_set1_RSA: function(pkey: PEVP_PKEY): TIdC_INT cdecl = nil;

where is the RSA Key? You can't use this function without this parameter, and I don't understand your comment. This is the correct header that actually makes signing possible:

EVP_PKEY_set1_RSA: function(pkey: PEVP_PKEY; key : PRSA): TIdC_INT cdecl = nil;

• EVP_DigestSignInit & EVP_DigestSignFinal are wrong

the openSSL doco:

int EVP_DigestSignInit(EVP_MD_CTX *ctx, EVP_PKEY_CTX **pctx,
                        const EVP_MD *type, ENGINE *e, EVP_PKEY *pkey);
 int EVP_DigestSignFinal(EVP_MD_CTX *ctx, unsigned char *sig, size_t *siglen);

your header:

  EVP_DigestSignInit: function(ctx: PEVP_CIPHER_CTX; pctx: PPEVP_PKEY_CTX; const type_: PEVP_MD; e: PENGINE; pkey: PEVP_PKEY): TIdC_INT cdecl = nil;
  EVP_DigestSignFinal: function(ctx: PEVP_CIPHER_CTX; sigret: PByte; siglen: PIdC_SIZET): TIdC_INT cdecl = nil;

Correct header:

  EVP_DigestSignInit: function(ctx: PEVP_MD_CTX; pctx: PPEVP_PKEY_CTX; const &type: PEVP_MD; e: PENGINE; pkey: PEVP_PKEY): TIdC_INT cdecl = nil;
  EVP_DigestSignFinal: function(ctx: PEVP_MD_CTX; sigret: PByte; siglen: PIdC_SIZET): TIdC_INT cdecl = nil;

It's the type of the first parameter

[grahamegrieve]

And @mezen you did not add the fips loading routines?

[rlebeau]

And @mezen you did not add the fips loading routines?

@grahamegrieve AFAIK, FIPS support is not available in OpenSSL 1.1.x, that will come in OpenSSL 3.0.

https://www.openssl.org/docs/fips.html

https://wiki.openssl.org/index.php/FIPS_modules

https://keypair.us/2020/10/openssl-3-0-fips-timeline/

[grahamegrieve]

Well, at least some of it is working for me now. I use and test:

IsHMACAvail := OpenSSLIsHMACAvail; IsHMACSHA256Avail := OpenSSLIsHMACSHA256Avail; GetHMACSHA256HashInst:= OpenSSLGetHMACSHA256Inst; UpdateHMACInst := OpenSSLUpdateHMACInst; FinalHMACInst := OpenSSLFinalHMACInst;

[mezen]

@grahamegrieve Ok, I was searching for a case statement. The two types of wrong header translation I did not see yesterday, after my working day. So it is always helpful if the error description is detailed :) I did not add your FIPS code to the loader, because it is not responsible for it. The Loader loads the function pointers into the variables. It is not responsible for more. But nobody prevents you from creating your own FIPS unit using loaded methods.

[grahamegrieve]

The Loader loads the function pointers into the variables. It is not responsible for more

the old one did. I already made my own but it seems appropriate for it to be somewhere common. @rlebeau?

thanks for the other fixes, and my apologies for not describing them better. Only issue left: figuring out the problem with http SSL Server

[JedrzejczykRobert]

@mezen

now when compile using D2007 I have:

[DCC Error] IdOpenSSLHeaders_ec.pas(290): E2003 Undeclared identifier: 'Pec_key_st' [DCC Error] IdOpenSSLExceptions.pas(239): E2003 Undeclared identifier: 'TStringArray'

D2007 not have TStringArray but have TStringDynArray

[mezen]

@JedrzejczykRobert that is the problem with developing for an old version which you dont have for testing :( Please try now again.

[JedrzejczykRobert]

@mezen

No problem, now I have installed D4, D7, D2007, XE7, XE10.4.1, so I can do tests :)

D7, D2007 -> compiled OK

XE10.4.1:

[dcc32 Error] IdOpenSSLExceptions.pas(169): E2066 Missing operator or semicolon [dcc32 Error] IdOpenSSLExceptions.pas(169): E2066 Missing operator or semicolon [dcc32 Error] IdOpenSSLExceptions.pas(169): E2014 Statement expected, but expression of type 'Pointer' found

the problem is:

Self.Create(AMsg) at

I comment code and use only IndyRaiseOuterException(Self.Create(AMsg));

[ralfjunker]

the problem is:

Self.Create(AMsg) at

Fix: raise Self.Create(AMsg) at

[mezen]

Fix: raise Self.Create(AMsg) at

Of course, the raise is gone missing by @JedrzejczykRobert zip file