Various non-valid Julia code output

[fredrikekre]

I recently used Clang.jl to generate bindings for HYPRE: (https://github.com/fredrikekre/HYPRE.jl). I had to add some things to the ignore list, where Clang.jl generated invalid Julia code, see: https://github.com/fredrikekre/HYPRE.jl/blob/373b9d7d5183bf8ddfa05af68d2ff00b5d98cc28/gen/generator.toml#L6-L27

Perhaps these are known issues, and if so please close, but thought I should atleast report it.

[Gnimuc]

I got the following errors when running the generator script:

➜  gen git:(master) ✗ julia --project generator.jl
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_config.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_utilities.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/10a7002eea557072e8e2ec81f16de1421d5bf667/include/mpi.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/10a7002eea557072e8e2ec81f16de1421d5bf667/include/mpi_proto.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/10a7002eea557072e8e2ec81f16de1421d5bf667/include/mpio.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_seq_mv.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_lobpcg.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_krylov.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/fortran_matrix.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/multivector.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/interpreter.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/temp_multivector.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_MatvecFunctions.h
[ Info: Parsing headers...
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:23:4: error: unknown type name 'MPI_Comm'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:25:4: error: unknown type name 'HYPRE_BigInt'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:26:4: error: unknown type name 'HYPRE_BigInt'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:28:4: error: unknown type name 'HYPRE_Int'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:33:4: error: unknown type name 'HYPRE_Int'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:36:4: error: unknown type name 'HYPRE_BigInt'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:37:4: error: unknown type name 'HYPRE_BigInt'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:38:4: error: unknown type name 'HYPRE_BigInt'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:39:4: error: unknown type name 'HYPRE_BigInt'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:40:4: error: unknown type name 'HYPRE_Int'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:41:4: error: unknown type name 'HYPRE_Int'
/Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h:67:15: error: unknown type name 'HYPRE_MemoryLocation'
[ Info: Processing header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE.h
[ Info: Processing header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/IJ_matrix.h
[ Info: Processing header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_IJ_mv.h
[ Info: Processing header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_parcsr_mv.h
[ Info: Processing header: /Users/gnimuc/.julia/artifacts/22c4fdd56b1e3cba1a0bf27bcc7884273ec7bc5d/include/HYPRE_parcsr_ls.h

It appears that IJ_matrix.h can't be correctly parsed because of unknown types. But Clang has a "feature" that if it doesn't recognize a type, it just treats the type as int. That's why the generated output contains a bunch of wrong Cint types.

After looking into the source code a bit, it turns out that IJ_matrix.h doesn't have any #include directives. This is probably just a miss. If this is intended then this header is not supposed to be used independently.

For example, some C APIs provide a macro for including all needed headers, and users need to always add this macro before including any headers in their source file.

However, in Clang.jl, each input header in headers will be independently parsed to a translation unit, so we need to patch the source code by adding another compile flag "--include=HYPRE_utilities.h"(You probably need to double-check which header is needed).

[fredrikekre]

Thanks, for the input. I actually don't see those errors ([email protected] with Julia 1.7.2). However, it looks like I don't need that header explicitly anyway.

Regarding the overflow of Int32, that seems to come from mpi.h ([email protected]+4) where we have

typedef int MPI_Datatype;

#define MPI_FLOAT_INT         ((MPI_Datatype)0x8c000000)

which Clang.jl translates to

const MPI_Datatype = Cint

const MPI_FLOAT_INT = MPI_Datatype(0x8c000000)

but shouldn't it be

const MPI_FLOAT_INT = 0x8c000000 % MPI_Datatype

instead?

[Gnimuc]

For this case, as there is no rule to one-to-one map a C-style cast to Julia, the result needs to be patched manually.

You could ignore that macro and put the right version in prologue.jl and add prologue_file_path = "./prologue.jl" in generator.toml.

[fredrikekre]

Okay thanks.

A couple of other things from mpi.h:

#define MPI_T_ENUM_NULL         ((MPI_T_enum)NULL)

becomes

const MPI_T_ENUM_NULL = MPI_T_enum(NULL)

which should be C_NULL I think? And

#define MPI_ARGV_NULL (char **)0

becomes

const MPI_ARGV_NULL = (Cchar * (*))(0)

which Julia doesn't even parse. Should be Ptr{Ptr{UInt8}}(C_NULL) maybe?

[Gnimuc]

const MPI_T_ENUM_NULL = MPI_T_enum(NULL)

I'm not sure, you probably need to check that library to see what an ENUM_NULL means.

const MPI_ARGV_NULL = (Cchar * (*))(0)

This is indeed a failure case of the current assumption used in Clang.jl.

For all of those macro issues, I recommend adding them to the macro ignore list and then manually patching them in prologue.jl.

[fredrikekre]

Okay thanks!

[Gnimuc]

I'd like to keep this open for tracking those failing cases.

[vchuravy]

I have another one:

#define FLUX_MSGHANDLER_TABLE_END { 0, NULL, NULL, 0 }

leads to:

const FLUX_MSGHANDLER_TABLE_END = {0, NULL, NULL, 0}