openwall
Naming of fallback john-* binaries
Hello, there is some discrepancy between the naming conventions used for the john packages/releases. As we are starting discussions about stream-lining the "make install" it might be beneficial to discuss and take the decisions on what the naming should be.
We has some discussion about that with @solardiz over emails and concluded this will be probably also interesting topic to @claudioandre-br and possibly others.
Currently there exists more than 3 naming conventions for john binaries : 1) john-CPUFEATURE / john-omp-CPUFEATURE - used mainly by the john core - default john-sse2 is without OMP, - the binary with OMP has -omp- in the name - suggested in param.h - john-omp mentioned in run/john.bash_completion, doc/FAQ:OpenMP-enabled, doc/README:choose - use https://github.com/openwall/Owl/blob/main/packages/john/john.spec
2) john-CPUFEATURE / john-CPUFEATURE-non-omp - used by jumbo where OMP is enabled by default john-CPUFEATURE (ex: john-sse2) binary - binaries wihtout OMP have extension -non-omp - params.h, doc/README-DISTROS mentions john-non-omp - packaging/build.sh using john-CPUFEATURE-non-omp - suggested on https://github.com/openwall/john/wiki/Fallback-binary-chains - it is bit strange to see in name what feature is not enabled as there might be probably more features missing, like opencl
3) john-non-CPUFEATURE / john-non-omp-non-CPUFEATURE - example: john-non-avx512f, john-non-avx512bw, john-non-mmx, john-non-omp-non-avx, - mic.h, params.h, x86-64.h, x86-sse.h - given probably just as example - have not seen this really used in some packaging, as it is bit unpractical to list features missing rather than features present
4) john-OS-ARCH-CPUFEATURE - for example john-macosx-x86-64 - John core was using Makefile targets in form of OS-ARCH-CPUFEATURE - for some (macos) targets in was moving files to run/ as - src/Makefile.in , src/Makefile.legac
It would be nice to consolidate that. What I am worried most is that john-CPUFEATURE (ex. john-sse2) can be with or without OMP, depending on where it came from, which might bring some unnecessary ambiguity.
Kali linux is dealing with this issue by adding -omp to the binaries built with OMP, but it's bringing some -omp redundancy in names: /usr/lib/john/john-avx-non-omp /usr/lib/john/john-avx-omp /usr/lib/john/john-avx2-non-omp /usr/lib/john/john-avx2-omp /usr/lib/john/john-avx512bw-non-omp /usr/lib/john/john-avx512bw-omp /usr/lib/john/john-avx512f-non-omp /usr/lib/john/john-avx512f-omp /usr/lib/john/john-base-non-omp /usr/lib/john/john-base-omp /usr/lib/john/john-xop-non-omp /usr/lib/john/john-xop-omp
Personally, I would rather see in name what is there and not what is missing, as we could be missing a lot of things/features. CPU features are also saying what is there and not what is missing. We are also not using the "john-non-avx512" binaries as suggested in the header files so I believe with same logic we should not be using "-non-omp". But if you decide to continue using "-non-omp".
Best regards Michal Ambroz
4) john-OS-ARCH-CPUFEATURE - for example john-macosx-x86-64
- John core was using Makefile targets in form of OS-ARCH-CPUFEATURE
- for some (macos) targets in was moving files to run/ as
This is only done in john-macosx-* targets, which are internal to Makefile (not listed in the help text) and are used by the macosx-universal target. The resulting binary from the latter target is simply john, and it has the other 3 binaries linked in. BTW, we could want to update that for currently relevant macOS archs (add arm64), if we're to resume using universal binaries on macOS at all. BTW, I don't recall why ppc64 is excluded from there, even though we do have separate macOS targets for it.
This macOS specific feature is orthogonal to fallback binaries - it's arch (such as ppc vs. x86) vs. instruction set extensions (such as AVX vs. AVX2). We could have fallback chains within each arch, so we could even have a top-level universal binary covering several archs and it would also possibly fallback to less capable binaries that are of the target arch (already determined by macOS when it invoked the right sub-binary in a universal binary). Then there's OpenMP vs. not. Oh wait, this suggests we would also prefer to include arch names in those fallback binary names, so that a non-OpenMP binary that doesn't have any special instruction set extension requirements doesn't have the same name when it's for different archs (which would be a problem if they're all in the same directory with the top-level universal binary)... or we could also use a universal binary for such fallback. This gets tricky.
I think it's tricky and obscure enough that we should mostly exclude it from our reasoning about the fallback binary naming, except in context of macOS universal binary building. If we ever revisit macOS universal binaries, we'd deal with this somehow - e.g., by using per-arch subdirectories for their fallback chains if there's a clash otherwise.
@claudioandre-br I notice that you're switching to positively named binaries (no "non-") in https://github.com/openwall/john-packages/pull/26 - great! I guess you'll update src/packaging/build.sh in here next? Thanks!
I guess you'll update
src/packaging/build.shin here next? Thanks!
Yes. Probably I'll remove it from here (it belongs to the *-packages repository).
I can remove it now or create a PR to do it now, but the renaming will happen in April.
Regarding src/packaging/build.sh:
Probably I'll remove it from here (it belongs to the
*-packagesrepository).
Personally I don't mind, but maybe distro packagers like @xambroz would have wanted something like it to stay? Also, don't we refer to it from the documentation?
I didn't find anything in the documentation (and it makes sense): it's more like a starting point or a blueprint for packagers.
In that case, keeping the .desktop and flatpak might also make sense.
@claudioandre-br moved away from the -non- naming in src/packaging/build.sh, however that naming will probably remain in default fallback filenames we have in code:
$ fgrep -e -non- *.h
params.h: * options like -DOMP_FALLBACK_BINARY='"john-non-omp-non-avx"' (leaving
params.h: * CPU_FALLBACK_BINARY at its default of "john-non-avx") or
params.h:#define OMP_FALLBACK_BINARY "john-non-omp"
x86-64.h:#define CPU_FALLBACK_BINARY "john-non-avx512bw"
x86-64.h:#define CPU_FALLBACK_BINARY "john-non-avx512"
x86-64.h:#define CPU_FALLBACK_BINARY "john-non-avx2"
x86-64.h:#define CPU_FALLBACK_BINARY "john-non-xop"
x86-64.h:#define CPU_FALLBACK_BINARY "john-non-avx"
x86-64.h:#define CPU_FALLBACK_BINARY "john-non-sse4.2"
x86-64.h:#define CPU_FALLBACK_BINARY "john-non-sse4.1"
x86-64.h:#define CPU_FALLBACK_BINARY "john-non-ssse3"
x86-mmx.h:#define CPU_FALLBACK_BINARY "john-non-mmx"
x86-sse.h:#define CPU_FALLBACK_BINARY "john-non-avx512bw"
x86-sse.h:#define CPU_FALLBACK_BINARY "john-non-avx512"
x86-sse.h:#define CPU_FALLBACK_BINARY "john-non-avx2"
x86-sse.h:#define CPU_FALLBACK_BINARY "john-non-xop"
x86-sse.h:#define CPU_FALLBACK_BINARY "john-non-avx"
x86-sse.h:#define CPU_FALLBACK_BINARY "john-non-sse4.2"
x86-sse.h:#define CPU_FALLBACK_BINARY "john-non-sse4.1"
x86-sse.h:#define CPU_FALLBACK_BINARY "john-non-ssse3"
x86-sse.h:#define CPU_FALLBACK_BINARY_DEFAULT "john-non-sse"
When those defaults are used, the build simply does not know what other builds are available - it only knows what CPU feature it found lacking, and that fallback was requested - hence the naming.
So unless and until we drop this functionality (ability to enable fallback without explicitly specifying a fallback binary filename), the -non- naming should stay at least there.
It's also still in several documentation files, but that can be changed without dropping functionality.