change Ptr to Ref in ccall syntax arguments

[vtjnash]

in v0.4, Ref is intended to be a more-general, direct replacement for Ptr as a ccall argument type. i believe this transformation should be completely safe and transparent to the user, but allow for greater generality for a function's inputs

[ihnorton]

How do we support 0.3 and 0.4 in the same codebase?

[bjarthur]

currently arguments that are pointers are simply not typed. so

int32 __CFUNC DAQmxGetSysNIDAQMajorVersion(uInt32 *data);

becomes

function DAQmxGetSysNIDAQMajorVersion(data)
    ccall((:DAQmxGetSysNIDAQMajorVersion,NIDAQmx),int32,(Ptr{uInt32},),data)
end

fortunately, one can checkout clang 34da43c656f8a2451c2f7d63b38a5cc62f22f15a so that it becomes

function DAQmxGetSysNIDAQMajorVersion(data::Ptr{uInt32})
    ccall((:DAQmxGetSysNIDAQMajorVersion,NIDAQmx),int32,(Ptr{uInt32},),data)
end

from there one can simply search on Ptr and replace with Ref. sure would be nice to see this fixed though.

[Gnimuc]

Do we still want this? what's the advantage of changing Ptr to Ref?

[ihnorton]

what's the advantage of changing Ptr to Ref?

Had to refresh about this myself because it's been a while -- according to the manual, the difference is that Ref is GC-preserved during the ccall. guess this would be especially important for code that calls back in to Julia via cfunction callbacks and could potentially trigger GC. So probably a good idea.

[Gnimuc]

IIUC, by using Ref, the explicitly GC.@preserve in the following example can be eliminated?

jlfuncptr = cfunction(xxx)
GC.@preserve jlfuncptr ccall(xxx,xxx,(xxx, Ptr{Cvoid}, xxx), xxx, jlfuncptr, xxx)

[yuyichao]

the difference is that Ref is GC-preserved during the ccall. guess this would be especially important for code that calls back in to Julia via cfunction callbacks and could potentially trigger GC. So probably a good idea.

No there's no such difference

[yuyichao]

IIUC, by using Ref, the explicitly GC.@preserve in the following example can be eliminated?

And you don't need that explicit GC.@preserve.

[Gnimuc]

Then is it necessary when jlfuncptr is unsafe_convert from a Base.Function to Ptr{Cvoid}? Could you elaborate more about the mechanism under the hood?

[yuyichao]

Please don't just remember these special cases, that's why people gets it wrong....................

The only thing to know is that the only guaranteed local mechanism to preserve an object from GC is GC.@preserve and the return value from cconvert in a ccall, so when you are using a normal ccall, you NEVER need to preserve the ccall argument assuming the conversion methods are implemented correctly. Ptr and Ref makes no difference in this regard. CFunction has nothing special in this regard.

You only need GC.@preserve when it's not used as ccall argument. This including converting to Ptr manually. CFunction is only special in that it can could be used as the callee in ccall in which case you must do the conversion yourself. It still has nothing special semantically though.

Now back to the issue, the only difference between Ref and Ptr are what Jameson said above,

Ref is intended to be a more-general

which effectively means that for all cases implemented in Base, you can replace the argument type from Ptr to Ref and Ref implements a few more convinent conversion rules that are not possible with Ptr. However, since it is not done in/before 0.4, doing it at this time is breaking. Custom unsafe_convert and cconvert can already depend on the current behavior.

And just to clarify, which I think is done in other issue but not here,

currently arguments that are pointers are simply not typed. so

Is the correct behavior.

so that it becomes

function DAQmxGetSysNIDAQMajorVersion(data::Ptr{uInt32})

was a regression.

[Gnimuc]

Thanks for the detailed explanation! I wish I could have read this earlier, am going to fix the misuse of GC.@preserve in my code.

[ihnorton]

@yuyichao you will probably save yourself a lot of time if you go fix the docs.

The memory is managed by Julia and the Ref signature alerts Julia's garbage collector to keep managing the memory for result_array while the ccall executes. If Ptr{Cdouble} were used instead, the ccall may still work, but Julia's garbage collector would not be aware that the memory declared for result_array is being used by the external C function. As a result, the code may produce a memory leak if result_array never gets freed by the garbage collector, or if the garbage collector prematurely frees result_array, the C function may end up throwing an invalid memory access exception.

[yuyichao]

I have commented many times that the particular doc you quote is actively misleading but I do not know what people are confused about causing such "clarification" to be added so I don't know what to do other than removing it. People that had once confused about it are in much better position to fix it.

[ihnorton]

Removing that section would help, yes.

[kmsquire]

Can someone here go to https://github.com/JuliaLang/julia/pull/31700 and point out more specifically which the section of the docs that should be removed (or suggest a modification)? Thanks.