Move more DES-based formats to bitslice DES code

[solardiz]

We currently use bitslice DES in these formats:

$ fgrep -l DES_bs.h *_fmt*.c
BSDI_fmt.c
DES_fmt.c
LM_fmt.c
MSCHAPv2_bs_fmt_plug.c
NETNTLM_bs_fmt_plug.c
opencl_DES_fmt_plug.c
trip_fmt.c

Two of them were moved to bitslice DES due to work by Deepika (thanks!)

$ fgrep -w Deepika *.c
DES_bs.c: * Deepika Dutta Mishra <dipikadutta at gmail.com> in
DES_bs_b.c: * Deepika Dutta Mishra <dipikadutta at gmail.com> in 2012, no
MSCHAPv2_bs_fmt_plug.c: * Modified for using Bitsliced DES by Deepika Dutta Mishra
NETNTLM_bs_fmt_plug.c: * Modified for using Bitsliced DES by Deepika Dutta Mishra

However, many of jumbo-specific formats continue to use OpenSSL's DES:

$ fgrep -l /des.h *_fmt*.c
KRB4_fmt_plug.c
KRB5_fmt_plug.c
NETLM_fmt_plug.c
NETSPLITLM_fmt_plug.c
as400_des_fmt_plug.c
dmg_fmt_plug.c
dpapimk_fmt_plug.c
itunes_fmt_plug.c
iwork_fmt_plug.c
keychain_fmt_plug.c
mozilla_ng_fmt_plug.c
ntlmv1_mschapv2_fmt_plug.c
o10glogon_fmt_plug.c
o3logon_fmt_plug.c
opencl_dmg_fmt_plug.c
opencl_gpg_fmt_plug.c
opencl_keychain_fmt_plug.c
opencl_zip_fmt_plug.c
oracle_fmt_plug.c
pem_fmt_plug.c
racf_fmt_plug.c
ssh_ng_fmt_plug.c
vnc_fmt_plug.c

Also, AFS_fmt.c inherited from core tree uses JtR's own non-bitslice DES code (only), but that's a relatively obscure format, so I don't care much. And KRB4_fmt_plug.c above appears to wrongly include DES_std.h, while not using anything from it (instead only using OpenSSL's DES) - if confirmed, we should drop that line to avoid further confusion. Other uses of DES_std.h in the list below are probably OK for now (they're special cases within formats that mainly use bitslice) - although we might get rid of some of them later as well:

$ fgrep -l DES_std.h *_fmt*.c
AFS_fmt.c
BSDI_fmt.c
DES_fmt.c
KRB4_fmt_plug.c
MSCHAPv2_bs_fmt_plug.c
NETNTLM_bs_fmt_plug.c
trip_fmt.c

We should identify non-obscure formats that still use solely/primarily non-bitslice DES where that matters for their overall performance, and proceed to move them to bitslice, on CPU for now. This should be easier than it used to be now that Deepika paved the way.

I think the Oracle (possibly including o*logon?), VNC, RACF, AS400 formats may be high on our list to switch to bitslice. (I was surprised to find Oracle wasn't switched to bitslice yet. IIRC, this was one of those Deepika worked on, but perhaps didn't finish the work? Or did we fail to merge a patch?)

A related issue is OpenSSL's dropping of DES support. This may force us to do something about uses of OpenSSL's DES code even where it doesn't matter for performance.

The above lists are for format files, but there are also uses in maybe-common files:

$ fgrep -l /des.h *.c | fgrep -v fmt
KRB4_std_plug.c
gpg_common_plug.c
mdc2dgst_plug.c

$ fgrep -l DES_std.h *.c | fgrep -v fmt
DES_bs.c
DES_std.c
ztex_descrypt.c

I think these are OK for now, but we'll need to revisit the first of these lists when dealing with OpenSSL's dropping of DES support.

[magnumripper]

I was surprised to find Oracle wasn't switched to bitslice yet. IIRC, this was one of those Deepika worked on, but perhaps didn't finish the work? Or did we fail to merge a patch?

I'm not aware of any posted work that we didn't merge. I guess she never finished it (if she even started).

[exaghost]

Hi! Do You (the project) has any update on this? Some formats are still out there in the wild. They are still as slow, if we are in 2008. :D Not to mention, OpenSSL 1.1.1 will be EoL this September.

[claudioandre-br]

Not to mention, OpenSSL 1.1.1 will be EoL this September.

I guess you are complaining that the existing Windows package here uses OpenSSL 1.1.1. All john packages use stable released libraries and OpenSSL 1.1.1 is what cygwin itself currently deploys.

There is no other released version by cygwin.

[edited]

And they are aware of what is happening upstream.

OpenSSL library version: 01010114f
OpenSSL 1.1.1t  7 Feb 2023

Still evolving:

OpenSSL 1.1.1u  30 May 2023

[solardiz]

Do You (the project) has any update on this?

Unfortunately, no. I'm very familiar with our bitslice DES CPU code (having been the one to introduce it and update it not too long ago for the formats where we do use it), but I am too busy with other stuff now. Do you want to contribute this and have other relevant basic skills? I could possibly help as a mentor. If I find time to rework one of these formats as an example, would you take care of many more? No promises, though. I am in fact really busy these days.

As to concerns about OpenSSL dropping DES, we're fine even with 3.x for now. They did deprecate DES and introduce and fix relevant bugs, but overall it's still in there and works for us.

Separately, I think Cygwin are smart in staying with a stable branch for as long as they can. No need for new bugs.

[claudioandre-br]

FTR: cygwin is not alone. From FreeBSD 12 to FreeBSD 14 [1][2].

OpenSSL library version: 01010114f
OpenSSL 1.1.1t-freebsd  7 Feb 2023

[1] OpenSSL in base (as seen used by Google cloud BSD image). [2] Now. V. 14 can migrate to 3.x later, who knows.