john
john copied to clipboard
openwall
OpenCL kernel headers not found (macOS)
MacOS BigSur, AMD Radeon Pro 5700 XT 16 GB
% ./john --list=build-info Version: 1.9.0-jumbo-1+bleeding-b49c763c4 2021-06-24 21:32:53 +0200 Build: darwin20.5.0 64-bit x86_64 AVX2 AC SIMD: AVX2, interleaving: MD4:4 MD5:5 SHA1:2 SHA256:1 SHA512:1 CPU tests: AVX2 $JOHN is ./ Format interface version: 14 Max. number of reported tunable costs: 4 Rec file version: REC4 Charset file version: CHR3 CHARSET_MIN: 1 (0x01) CHARSET_MAX: 255 (0xff) CHARSET_LENGTH: 24 SALT_HASH_SIZE: 1048576 SINGLE_IDX_MAX: 2147483648 SINGLE_BUF_MAX: 4294967295 Effective limit: Number of salts vs. SingleMaxBufferSize Max. Markov mode level: 400 Max. Markov mode password length: 30 clang version: 12.0.5 (clang-1205.0.22.11) (gcc 4.2.1 compatibility) OpenCL headers version: 1.2 Crypto library: OpenSSL OpenSSL library version: 0101010bf OpenSSL 1.1.1k 25 Mar 2021 File locking: fcntl() fseek(): fseek ftell(): ftell fopen(): fopen memmem(): System's times(2) sysconf(_SC_CLK_TCK) is 100 Using times(2) for timers, resolution 10 ms HR timer: mach_absolute_time(), latency 33 ns Total physical host memory: 72 GiB Available physical host memory: 56299 MiB Terminal locale string: UTF-8 Parsed terminal locale: UTF-8
Path error:
% ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ./opencl/pbkdf1_hmac_sha1_kernel.cl
Build log:
Error building kernel ./opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522 0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1292 - clBuildProgram
I tried running john from a different path, no luck.
I specified the full path to the include instruction but still the file wasn't found.
Off the top of my head, to get around this you need to run john (at least once) from inside the opencl/ folder.
cd opencl
../john --test --format=opencl
Well, maybe you need to find the "right location" (if the 'opencl' folder does not fix it), as stated here https://www.openwall.com/lists/john-users/2014/08/03/2 (notice, this is the old headers location).
"OS X also has a problem with run-time compile of kernels that include header files. A workaround is to cd to the src directory and run each OpenCL format once. After that, the kernel binary is cached so you can move away from the src directory."
So the OpenCL compiler somehow ignored -I opencl that was passed to it. I wonder why this issue keeps occasionally reoccurring on various systems. Is there anything we're doing wrong (I don't think so)? Is there anything we can do better (e.g., maybe -I . along with opencl/ in paths inside the #include directives would have worked better across systems)?
@magnumripper Don't you use a similar Mac system now, yet aren't running into this issue?
@claudioandre-br tried that (per that post, it mentions the "kernels" directory that is not there anymore, replaced by opencl, so I already did that) with no luck.
The odd part is that I modified the #include's to match my exact location of the .h files (full path) and it still says the file is not found.
This is a very old problem, I tried all the tricks from part workaround and none of them worked. Also, the kernels directory was moved to the opencl directory.
If I can help compiling and test any changes, count on me.
Thanks!
On 2021-06-25 14:39, Solar Designer wrote:
So the OpenCL compiler somehow ignored
-I openclthat was passed to it. I wonder why this issue keeps occasionally reoccurring on various systems. Is there anything we're doing wrong (I don't think so)? Is there anything we can do better (e.g., maybe-I .along withopencl/in paths inside the#includedirectives would have worked better across systems)?@magnumripper Don't you use a similar Mac system now, yet aren't running into this issue?
I haven't heard of that issue for years. If I ever did see it myself I'm sure I committed workarounds that should still be in place.
magnum
What if you move all files in run/opencl to just run? That'd be ugly but it would be interesting to see if it matters.
Moved everything to the run/opencl folder and made the file structure also:
% pwd
[REDACTED]/Tools/john/run/opencl
% ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Can't read source kernel: No such file or directory
That's the output.
What if you move all files in
run/openclto justrun? That'd be ugly but it would be interesting to see if it matters.
Copying (not moving). And enable OpenCL verbose (the way it works in MacOS):
- To check if the Mac is compiling the kernel in some weird folder like
/temp/cl88ddSdoui(it used to, right?) - Well, hope we can see something
newin the outputs.
Hi @claudioandre-br and @magnumripper !
I don't really know how to enable verbosity for OpenCL in MacOS. I Googled it but didn't find anything useful. If you can help on that, I can try it.
I compiled a fresh copy and now instead of moving I copied the needed files to the run/opencl folder. Same error.
Running John binary from different locations:
moebius@kandinsky opencl % ../john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ../opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
^
Error building kernel ../opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1292 - clBuildProgram
moebius@kandinsky opencl % ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Can't read source kernel: No such file or directory
Thanks!
What if you move all files in
run/openclto justrun? That'd be ugly but it would be interesting to see if it matters.Copying (not moving). And enable OpenCL verbose (the way it works in MacOS):
Right, also move the headers to run, not the CPU binaries to opencl. From a normal tree it'd be like cd run; cp opencl/*.h .. However, it wont solve the below:
- To check if the Mac is compiling the kernel in some weird folder like
/temp/cl88ddSdoui(it used to, right?)
Ah yes this sounds familiar. I fixed it at some point by starting to pass -I /full/path/to/john/opencl to the kernel runtime build instead of just eg. -I opencl but apparently that doesn't seem to kick in according to OP. Did we break it recently?
I wonder why I'm not seeing those problems at all? I run Big Sur on intel and M1 macbooks.
Ah yes this sounds familiar. I fixed it at some point by starting to pass
-I /full/path/to/john/openclto the kernel runtime build instead of just eg.-I openclbut apparently that doesn't seem to kick in according to OP. Did we break it recently?
Looks like @solardiz broke it in 753dd9f102.
We could want to revert 753dd9f10 but change the inital ifdefs to eg, #if __APPLE__
-#if !defined(__CYGWIN__) && !defined(__MINGW32__)
+#if __APPLE__
+// If true, use realpath(3) for translating eg. "-I./opencl" into an absolute
+// path before submitting as JIT compile option to OpenCL.
+#define I_REALPATH 1
+#endif
Why am I not affected by these problems though? I'm not sure I ever was, using Macs since 2012 or so.
Would we also avoid the issue by chdir to opencl/ and passing -I .? If so, I think that's the way to go, and we could do it unconditionally on all OSes.
@magnumripper I've been using John in Macs since 2008 and this is also the first time. I moved from an old iMac (2010, ATI reader Graphics Card) to this new one, and I found this error when doing the compilation process and some benchmarks. First time too, I can state that OpenCL was running OK on the old iMac. Maybe the old MacOS is doing something different?
Let me know if I can help testing any changes on the code.
Thanks.
Would we also avoid the issue by
chdirtoopencl/ and passing-I .?
No - as Claudio reminded us, some versions of MacOS and/or drivers will copy the kernel file (supplied as a char array IIRC) to some temporary directory. I can't see any way to fix the issue except supplying a full path to our opencl directory. Well, unless we do a cpp stage ourselves... that'd be a nice can of worms, LOL...
I assume this only happens with particular combos of devices and driver versions and that would explain why I never had the problem myself (or TBH maybe I did have it at some point long ago and don't even remember).
We could want to revert 753dd9f but change the inital ifdefs to eg,
#if __APPLE__-#if !defined(__CYGWIN__) && !defined(__MINGW32__) +#if __APPLE__ +// If true, use realpath(3) for translating eg. "-I./opencl" into an absolute +// path before submitting as JIT compile option to OpenCL. +#define I_REALPATH 1 +#endifWhy am I not affected by these problems though? I'm not sure I ever was, using Macs since 2012 or so.
I vote to apply (something like) this:
- Nobody is going to create and test a general solution for Windows and Mac.
@claudioandre-br if you patch something, I can pull the code and test it in my MacOS. I can try an older version too... thanks!
@cintainfinita try using latest & greatest code, then just git revert 753dd9f. Then build and test and please report back.
@magnumripper
% git revert 753dd9f
Auto-merging src/opencl_common.c
[bleeding-jumbo 48a953279] Revert "OpenCL: Build kernels having chdir'ed to John's home directory"
Committer: moebius <[email protected]>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:
git config --global --edit
After doing this, you may fix the identity used for this commit with:
git commit --amend --reset-author
1 file changed, 109 insertions(+), 45 deletions(-)
No luck, same error:
moebius@kandinsky opencl % ../john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I /Users/moebius/Documents/Tools/john/run/opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ../opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
^
Error building kernel ../opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1356 - clBuildProgram
moebius@kandinsky opencl % cd ..
moebius@kandinsky run % ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I /Users/moebius/Documents/Tools/john/run/opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ./opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
^
Error building kernel ./opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1356 - clBuildProgram
This seems to be a different problem than the "temporary directory" one. I have no idea what is happening there. Maybe try building latest hashcat and see if you get the same problem with that?
Would we also avoid the issue by
chdirtoopencl/ and passing-I .? If so, I think that's the way to go, and we could do it unconditionally on all OSes.
Maybe we should actually try this here.
Hi @magnumripper.
Look!
% ./hashcat -D 1 -b
hashcat (v6.2.2-70-g5bea13200) starting in benchmark mode
Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.
OpenCL API (OpenCL 1.2 (May 8 2021 03:14:28)) - Platform #1 [Apple]
====================================================================
* Device #1: Intel(R) Core(TM) i9-10910 CPU @ 3.60GHz, 73664/73728 MB (18432 MB allocatable), 20MCU
* Device #2: AMD Radeon Pro 5700 XT Compute Engine, skipped
Benchmark relevant options:
===========================
* --opencl-device-types=1
* --optimized-kernel-enable
Hashmode: 0 - MD5
clCompileProgram(): CL_BUILD_PROGRAM_FAILURE
<program source>:7:10: fatal error: 'inc_vendor.h' file not found
#include "inc_vendor.h"
^
* Device #1: Kernel /Users/moebius/Documents/Tools/hashcat/OpenCL/shared.cl build failed.
Started: Tue Jun 29 09:55:04 2021
Stopped: Tue Jun 29 09:55:04 2021
Looks like hashcat is "affected" too. I will see if I can find anything useful here for John.
Update on hashcat:
1)I saw someone stating that installing hashcat using brew solved the problem. And it worked. hashcat worked.
- sudo make install on the hashcat code. run it outside the hashcat's directory. worked!
installing hashcat using brew solved the problem
That's odd, I can't see how/why it would matter. How did you build John? If you didn't use brew, maybe try the macOS instructions at bottom of doc/INSTALL:
1. Install Xcode (free in App Store).
2. Install Xcode's command-line tools:
xcode-select --install
You'll need to re-run that command after any little update of macOS,
because the command-line tools silently disappeared and you'll get a silly
"error: C compiler cannot create executables" when trying to build John.
3. Install Homebrew, by following the instructions given here:
https://brew.sh
4. Install Homebrew's gcc and OpenSSL:
brew install gcc openssl
5. Consider adding homebrew directories such as /usr/local/bin and/or
/opt/homebrew/bin to your $PATH.
6. Configure, possibly adding a CC option for pointing to a specific gcc and
using whatever LDFLAGS and CPPFLAGS was recommended when you installed
Homebrew's OpenSSL:
./configure CC="/opt/homebrew/bin/gcc-10" \
LDFLAGS="-L/opt/homebrew/opt/openssl/lib" \
CPPFLAGS="-I/opt/homebrew/opt/openssl/include"
7. Clean old files and make:
make -s clean && make -sj4
If this does solve the problem, I'm really stumped as of why!
From what I understand in https://github.com/hashcat/hashcat/issues/2270, the key for hashcat isn't really using brew, but running "make install". That makes even less sense.
We need to test this before digging more into this - @cintainfinita try applying this patch to a clean tree (EDIT: or pull #4762):
diff --git a/src/opencl_common.c b/src/opencl_common.c
index 17ff13871..42d8f090a 100644
--- a/src/opencl_common.c
+++ b/src/opencl_common.c
@@ -1146,7 +1146,7 @@ static char *get_build_opts(int sequential_id, const char *opts)
global_opts = OPENCLBUILDOPTIONS;
snprintf(build_opts, LINE_BUFFER_SIZE,
- "-I opencl %s %s%s%s%s%d %s%d %s -D_OPENCL_COMPILER %s",
+ "%s %s%s%s%s%d %s%d %s -D_OPENCL_COMPILER %s",
global_opts,
get_platform_vendor_id(get_platform_id(sequential_id)) ==
PLATFORM_MESA ? "-D__MESA__ " :
@@ -1228,12 +1228,7 @@ void opencl_build(int sequential_id, const char *opts, int save, const char *fil
#endif /* HAVE_MPI */
/*
- * Build kernels having temporarily chdir'ed to John's home directory.
- *
- * This lets us use simply "-I opencl" instead of having to resolve a pathname,
- * which might contain spaces (which we'd have to quote) and was problematic on
- * Cygwin when run from Windows PowerShell (apparently, with Cygwin resolving
- * pathnames differently than the OpenCL backend would for the includes).
+ * Build kernels having temporarily chdir'ed to John's opencl directory.
*
* Saving and restoring of the current directory here is incompatible with
* concurrent kernel builds by multiple threads, like we'd do with the
@@ -1247,18 +1242,18 @@ void opencl_build(int sequential_id, const char *opts, int save, const char *fil
int old_cwd_fd = -1;
char old_cwd[PATH_BUFFER_SIZE];
old_cwd[0] = 0;
- char *john_home = (char *)path_expand_safe("$JOHN/");
- if (john_home[0] && strcmp(john_home, "./")) {
+ char *john_opencl = (char *)path_expand_safe("$JOHN/opencl");
+ if (john_opencl[0] && strcmp(john_opencl, "./opencl")) {
old_cwd_fd = open(".", O_RDONLY);
if (!getcwd(old_cwd, sizeof(old_cwd))) {
old_cwd[0] = 0;
if (old_cwd_fd < 0)
fprintf(stderr, "Warning: Cannot save current directory: %s\n", strerror(errno));
}
- if (chdir(john_home))
- pexit("chdir: %s", john_home);
+ if (chdir(john_opencl))
+ pexit("chdir: %s", john_opencl);
}
- MEM_FREE(john_home);
+ MEM_FREE(john_opencl);
build_code = clBuildProgram(*program, 0, NULL, build_opts, NULL, NULL);
if ((old_cwd_fd >= 0 || old_cwd[0]) && /* We'll only have errno when we attempt a *chdir() here */
(old_cwd_fd < 0 || fchdir(old_cwd_fd)) && (!old_cwd[0] || chdir(old_cwd)))
It works just fine on my macBook (but that one didn't have problems before either). I opted to not include -I . on line 1149 because it really shouldn't be needed, but in case you still have problems with the above patch, try re-adding -I . to that line where it previously said -I opencl.
@solardiz on a side note I wonder whether using #include <opencl_device_info.h> would work better than #include "opencl_device_info.h" for the original code without the above patch... I'm thinking -I is normally used for finding <file.h> whereas "file.h" would be looked for in current directory, right?
Anyway, in case the above patch happens to work it's probably better (safer on other platforms).
EDIT: I just tried this for phun:
#include "stdio.h"
int main(int argc, char **argv)
{
while (argc--)
printf("%s ", *argv++);
printf("\n");
}
All compilers I tried finds stdio.h even though listed with quotes instead of <>. Some googling indicates this is normal manner - quotes just tries current dir before the others.
I'm thinking
-Iis normally used for finding<file.h>whereas"file.h"would be looked for in current directory, right?
I think you're right. We can try that.
Hi all! thanks for all your efforts!
-
Tried the patch without the -I. No luck, .h still not found. Added the -I. Same error.
-
I modified the .cl, changing "test.h" with <test.h>. Same result, file not found.
To compile john, I just used ./configure and then the suggested make commands. I have openssl headers, etc.
Maybe something related to GCC?, here's my output:
% gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
So, I brew installed GCC-10 and point the configure process to the new one: % ./configure CC="/usr/local/Cellar/gcc@10/10.3.0/bin/gcc-10"
% ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ./opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
^
Error building kernel ./opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1287 - clBuildProgram
Sorry, can't make it work. Let me know. Thanks!
It shouldn't be related at all to the host code's compiler or using brew or not. The thing we see failing is the OpenCL subsystem trying to build the kernel - and it's definitely using Apple's framework which we have little control over.
I see no way around this other than either just declaring any system behaving this way b0rken (as in not our problem) - or actually try a cpp pass ourselves. The latter would be pretty silly...