'gnutls_certificate_set_x509_key_file' and similar functions causes crash: An invalid parameter was passed to a function that considers invalid

[NevermindExpress]

When I call gnutls_certificate_allocate_credentials to a gnutls_certificate_credentials_t and then call gnutls_certificate_set_x509_key_file on the same struct, it crashes the program. I don't think it's a mainstream GnuTLS problem because exact same code works fine on Linux.

Your Environment

• Version Used: 3.7.8
• Operating System and Version: Windows 8.1 (NT 6.3.9600)
• Compiler and Version(s): MSVC 16 (VS 2019)

[NevermindExpress]

I compiled the libraries with debug configuration and it crashes on right here

Edit: I just found the "e" flag is specific to glibc and not a standard, and doesn't exist on VCRT. So that's what causes the crashes.

Edit 2: Forgot to say i already solved it by removing the "e" from flags but now there's loads of memory issues which makes it useless (and they are unrelated to this).

[Sibras]

Have you tried replacing the 'e' with 'N' for windows platforms and see if that helps.

If you have a small test program you can post that can be used to demonstrate the bug and test fixes then that would be useful to help incorporate a fix.

[NevermindExpress]

I tried deleting "e" and it solved the problem but i got lots of memory related issues (and they have no relation with that) which made whole library unusable. Not sure if that memory errors are my fault or a bug with the project but the crash with that particular thing is fixed by removing "e".

[ronchristie52]

Hi Sibras!

Just as an addition to this issue:

One of the functions that Harbour (the platform I'm using) implements is an fopen() call that mimics the standard windows/linux library fopen(cFile, cMode) call and the gnutls library also uses that same call, but the mode argument is different in the gnutls library call. The gnutls library has support for an extended mode string that in addition to the usual “r”, “w” and “a” arguments also has support for “b”, “x” and “e” arguments. As nearly as I can tell, these arguments are dealt with in the gnutls file fopen.c, (under the function rpl_fopen() but fopen.c doesn’t appear to be mentioned in the .vcproj file that SMP implements, and I'm not sure that I'm decoding the various .h files and macros that are involved in re-routing the gnutls fopen() function to rpl_fopen()

I’m an okay-ish C programmer, but not nearly good enough to decode the multiple layers of .h files, project files that seem to be involved in determining whether or not fopen.c should be included and what (if any) macros should be defined.

Could you take a look at this and advise me how I can get this issue resolved. Thanks for your help!

[Sibras]

Unfortunately its rather difficult to get it to work as gnulib requires to be able to hijack the system fopen with its own, but its own also requires to hijack other system functions (fdopen etc.) which makes it a bit complicated. The issue is that gnutls is using a gnulib submodule to define helper functions, as these are pulled in from a submodule they cant be directly modified by this project as otherwise I would just delete the "e" and call it a day. The "e" doesnt have the same meaning on windows and so can be just removed without issue which would be the easiest way to go about it.

[ronchristie52]

Hi Matthew!

Removing the "e" does resolve the specific error, but leaves other assertions in place, so gnutls is no longer usable in Windows. I'll look for a different solution - probably Botan to start.

Thanks for getting back to me.

[kervala]

I had this issue too :( I replaced these calls by gnutls_certificate_set_x509_trust_mem with a custom file loading and it worked fine :)

[kervala]

Finally, even though I patch that, nothing is working as expected (it hangs in handshake) :(

Please are these binaries of GNU TLS working for someone under Windows ? Thanks :)

[talregev]

@Sibras Any solution? curl was dropping shiftmedia-libgnutls vcpkg port from their ci because there is a broken tls.