kaby76
Compilation errors don't specify line/column numbers so it's hard to fix .PIG errors in code blocks
In a Piggy spec file (.PIG), code blocks could contains static and dynamic errors, like using bad C# syntax. Piggy collects all the code blocks into a C# file that it generates, formats the code, then runs the Roslyn compiler on that. Unfortunately, the Roslyn compiler seems to not like to give much information on where the compilation occurs. The location it gives for a syntax error is not in a line/col format; it's a straight character range of the error in the file from the start of the file. I need to output lines, errors, etc., showing where the error is. Otherwise, it's very hard to use.
I wonder if this is the problem I was running into because I would always get "missing ';' like this, when running piggy.dll.
...
var scope = _stack.Peek();
var name = tree.Attr("Name");
if (name == "") return;
if (scope.getSymbol(name) != null) return;
var type = new TypeAlias(name, null);
scope.define(type);
}}
))))
}
}<missing ';'><EOF>
There are actually two types of errors here. The first which you see is an Antlr parsing error. It's trying to parse the template file, but can't because it's missing a matching closing "}" or ")" or ";". So, I use ";" to close off a few things in a template file: after "using 'foobar.pig';", after "application ...;". Admittedly, the grammar could be improved. The other place you see errors is from the Roslyn compiler. After matching, the engine collects all those code blocks "{{ ... }}" and puts it in a file to compile. Those are the errors I was referring to earlier. The compiler just says the error is in the range "12353..12366". I have to convert that to a line/col number and probably just the source. BTW, Piggy isn't quite done, but it's getting better. I'm working on a Nuget package so you can just do a "build" and out pops the compiled interface. Check out the cuda.pig example, and you might look at the other templates. Currently, the generator doesn't work for unions, nor things like std::string, but that in due time.
I watch your repo and check for changes every couple of days. I recompile if the changes look interesting. One thing I don't want to do is recompile all the llvm stuff again. Wow, that was a pain. Not that it was hard but it took hours and hours.
IIRC i tried all the pig files except the cuda one for the repo I was trying to port. Every file gave a similar error as I posted above. So, I'll continue to wait. ;)
Yes, compiling llvm is a pain. I think I make use of code that isn't exported in the standard binaries release, like the astvisitor class. I did set up a mirror with the full libs and dlls at one point, but stopped. I will look around to see if there is another project is doing those builds so we don't have to. --Ken
@fdncred Check out the self-contained example for Piggy here: https://github.com/kaby76/PiggySimple. It's a start. But, there is still a lot to do.
Wow! Very nice. I made the following changes to the c# code so I can see the return string.
static void Main(string[] args)
{
var test1 = Functions.Test1();
var str1 = Marshal.PtrToStringAnsi(test1);
Functions.Test2(out IntPtr str);
var str2 = Marshal.PtrToStringAnsi(str);
var s1 = Functions.Test3();
Functions.Test4(out struct1 as1);
}
and in launchSettings.json - so, I could step into the cpp code from c#
{
"profiles": {
"Csharp": {
"commandName": "Project",
"nativeDebugging": true
}
}
}
I guess it's in one of the pig files that tells it that char * should be an IntPtr? I assume all the mess was in there. But i expected to see a bunch of DllImports as usual and there are none. This must be the magic of piggy. Good job!
It looks like project1.pig is doing the mapping work of "const char *" to "string", however, Functions.Test1() returns an IntPtr instead of a string. What am I missing?
Hi Darren, I'll add in those changes right away.
Yes, the "char *" is mapped into IntPtr through the templates. Take a look at the template file Funcs.pig in piggy. Then, after doing a build of the example, take a look at the project1.pig.cs file in the build output directory. In the PiggySimple example, the PIG file https://github.com/kaby76/PiggySimple/blob/master/Csharp/project1.pig sets up a dictionary for mapping C++ types into C# types. It does it through the "init" code block. If you look at the project1.pig.cs, that code turns into a constructor. In Funcs.pig, there's some C# code to pick off the parameter or function type--everything in the AST is just a C# string, e.g., "tree.Attr('name')"--and convert it to IntPtr (https://github.com/kaby76/Piggy/blob/master/Templates/Funcs.pig#L40). It's just a string Dictionary. Really, Piggy doesn't do that much. The tool itself reads the AST and the PIG files, matches patterns to the AST, generates the C# code that is in your templates, and calls that code with a node in the AST. The real guts of how it's converted is in the template code. The templates I give implement a p/invoke generator, but you can really do what ever you like in these templates.
There's still a lot to do with piggy, but it's starting to be usable now. I have it working, sort of, for NVIDIA's CUDA toolkit. But, it doesn't work for unions, and STL. I'm going to be fishing around in CppSharp, ClangSharp, and SWIG to see what basic type mapping is done there, and maybe pick up some of that code.
Ken
On 1/14/2019 12:57 PM, Darren Schroeder wrote:
Wow! Very nice. I made the following changes to the c# code so I can see the return string.
static void Main(string[]args) { var test1 = Functions.Test1(); var str1 = Marshal.PtrToStringAnsi(test1); Functions.Test2(out IntPtr str); var str2 = Marshal.PtrToStringAnsi(str); var s1 = Functions.Test3(); Functions.Test4(out struct1 as1); }
and in launchSettings.json - so, I could step into the cpp code from c#
{ "profiles": { "Csharp": { "commandName":"Project", "nativeDebugging":true } } }
I guess it's in one of the pig files that tells it that char * should be an IntPtr? I assume all the mess was in there. But i expected to see a bunch of DllImports as usual and there are none. This must be the magic of |piggy|. Good job!
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaby76/Piggy/issues/6#issuecomment-454099968, or mute the thread https://github.com/notifications/unsubscribe-auth/AK3KC98hpwNo9yfaX1tzMm1XI0Aqn44qks5vDMUPgaJpZM4ZpzgG.
Looks like I need to update the mappings in project1.pig for the function return. There are two "C to C#" mappings, one for parameters, the other for function returns. I'll update the example with a fix. Thanks. --Ken
On 1/14/2019 1:02 PM, Darren Schroeder wrote:
It looks like project1.pig is doing the mapping work of "const char *" to "string", however, Functions.Test1() returns an IntPtr instead of a string. What am I missing?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaby76/Piggy/issues/6#issuecomment-454101676, or mute the thread https://github.com/notifications/unsubscribe-auth/AK3KC_KdN5rf666Lc1_c9ti6Mdxu3BNZks5vDMY7gaJpZM4ZpzgG.
Is the ast output somewhere with simplepiggy? It might be helpful for debugging but I'm not sure. I can't seem to figure out why "const char *" isn't being translated to "string" even though it's in both the ModParamUsageType and MonNonParamUsageType.
Also, is there a comment character for your pig files so one can comment out specific lines and those lines be ignored by Antlr or whatever reads these files?
Hi Darren,
You bet. In Visual Studio, open the PiggySimple solution file. Then do a build. Next, in the Solution Explorer, click on the "Show All Files" icon at the top of that panel. It should show "bin", "obj", etc. Next, open up the path "obj/Debug/netcoreapp2.2". That's the output directory, and it should contain "project1.pig.cs". (Alternatively, you could just open Windows File Explorer to that directory.) That's the output of the PiggyTool, and it contains the p/invoke declarations for the functions and all the types they need. Another way to do this is to right-click on one of the method calls in Program.cs, like Test2, and "Go to definition". That will open up the file. Now, the AST itself is also contained in the obj/Debug/netcoreapp2.2 directory. Look for "ast.txt". The inputs to the tool are "project1.cpp" and "project1.pig", the same directory as Program.cs. They're source files, so that's why they're there. You can change where all this stuff goes by right clicking on "project1.pig" and going to Properties. There's a bunch of options there. Note, I think I need to add in double quotes to some of these file locations because I haven't check if all this work with paths that contain spaces.
--Ken
On 1/15/2019 8:53 AM, Darren Schroeder wrote:
Is the ast output somewhere with |simplepiggy|? It might be helpful for debugging but I'm not sure. I can't seem to figure out why "const char *" isn't being translated to "string" even though it's in both the ModParamUsageType and MonNonParamUsageType.
Also, is there a comment character for your pig files so one can comment out specific lines and those lines be ignored by Antlr or whatever reads these files?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaby76/Piggy/issues/6#issuecomment-454399140, or mute the thread https://github.com/notifications/unsubscribe-auth/AK3KCwuNg-R5j85haZs2U6txIumn9El-ks5vDd1CgaJpZM4ZpzgG.
Yes, I have seen the project1.pig.cs and even tried to change it and recompile to no avail. Ok, I see the obj/Debug/netcoreapp2.2/ast.txt. That's what I was looking for. I'll need to study it now.
I'm also thinking more about when I go to write my own pig file for processing leptonica. Troubleshooting errors seems difficult at this point. Maybe it's just because I'm confused why neither ModParamUsageType and ModNonParamUsageType seem to be working for "const char *". Something is going wrong somewhere because it's changing "const char *" to an "IntPtr" instead of a "string" and I'm not sure how to troubleshoot that.
I didn't even see the properties on the pig file. Thanks.
I've started working with Leptonica now. Lep. builds fine and looks good (as per cmake/msbuild for x64/Release build). The first real problem is that ClangSerializer needs to take multiple include directories (build/src and src). Ast.txt, the output of ClangSerializer, is basically empty. Running the ClangSerializer tool itself from a Bash shell gives the error, but in the VS Output window itself, VS just reports the build failed. So, two problems I need to fix and I can then continue with this example. I'll let you know as I progress today.
Wow! I didn't expect you to look at Leptonica. Thanks!
Using cmake, if you do a build install, it copies everything to the defined install folder. So, for me, that creates an include\leptonica folder with all the .h files. However, with NClang and ClangSharp I think I use that folder as well as getting the .c files from the src folder. I may even include .h,.c from the prog folder because there is useful code in there as well.
It may also be worth mentioning that Leptonica has a file called allheaders.h which, as you guessed it, is a concatenation of all the header files (I think). You may be able to start with only that file.
I guess allheaders.h is not a concatenation but the prototype file for all functions. So, it would contain the items that go in the C# DllImport lines.
You will have to include the .c files at some point because there are structs, enums, etc defined in those files.
Maybe I should just delete all these comments and say, Thanks, and you're right. ;)
Looks like ClangSerializer fails to work when generating the AST for "allheaders.h", so the PiggyTool itself cannot input/parse the AST file from the serializer. The weakest link in the system is Clang IMHO. Clang used to have an XML serializer, but apparently, it was always out of date. I took the code for Clang (which is why you see you have to build LLVM/Clang), then wrote a serializer from scratch. It was a royal pain to write, because Clang's ASTVisitor code is really poorly designed.
OK, I have most of Leptonica wrapping. But, there are two problems. One is that FILE isn't wrapped yet, and will need to figure this out because it's in C runtime DLL (you can't static link any of this). The other are some parameters marked "out" when they're really "in" because it's a pointer to a struct, eg pixMaskedThreshOnBackgroundNorm. I'll need to think about this. Otherwise, it's looking pretty good.
Please check out the piggy files for Leptonica. https://github.com/kaby76/Piggy/blob/master/include.c https://github.com/kaby76/Piggy/blob/master/Templates/lep.pig You're going to need a CSPROJ with some defaults for the PIG file for your C# code, this might help. ` <Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>netcoreapp2.2</TargetFramework> </PropertyGroup>
<ItemGroup> <PackageReference Include="piggy" Version="1.0.5"> <PrivateAssets>all</PrivateAssets> <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets> </PackageReference> </ItemGroup>
<ItemGroup> <Piggy Update="lep.pig"> <ClangSourceFile>include.c</ClangSourceFile> <AstOutputFile>ast.txt</AstOutputFile> <ClangOptions>"I../src" "I../build/src"</ClangOptions> </Piggy> </ItemGroup>
</Project>
`
ok, i have a new dotnet core console app created. I've changed my csproj to look like this.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.2</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="piggy" Version="1.0.5">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<Piggy Update="lep.pig">
<ClangSourceFile>include.c</ClangSourceFile>
<AstOutputFile>ast.txt</AstOutputFile>
<ClangOptions>"ID:/Src/GitHub/Leptonica/src" "ID:/Src/GitHub/Leptonica/build/src"</ClangOptions>
</Piggy>
</ItemGroup>
</Project>
I have added include.c and lep.pig to the solution.
I build and it waits for a while and then just says failed. I have validated that Piggy1.0.5 is part of the solution via Manage Nuget Packages in Visual Studio.
Do the include.c and the lep.pig need to be in a particular directory? I'm guessing it can't find them.
My detailed build log says this.
1>Target PiggyCompile:
1> Building target "PiggyCompile" completely.
1> Output file "obj\Debug\netcoreapp2.2\LeptPig.csproj.PiggyGeneratedCodeFileListAbsolute.txt" does not exist.
1> Using "PiggyClassGenerationTask" task from assembly "C:\Users\myusername\.nuget\packages\piggy\1.0.5\build\..\lib\netstandard2.0\Piggy.dll".
1> Task "PiggyClassGenerationTask"
1> Done executing task "PiggyClassGenerationTask" -- FAILED.
So, I'm not sure what is going on.
I just noticed that I do have an ast.txt file that is 6,298 kb. So, that part seems to be working.
The lep.pig needs to be updated. Let me explain how this works. ClangSerializer outputs the AST, including a ton of MS include files, like stdio.h, etc. In order to not wrap the entire world of MS include files, there's this "limit" variable you see in the PIG file. The templates pick up on that e.g., SrcRange=$"{Enums.limit}" in Enums.pig. Here, you need to override that, limit = "Src". The string $"{Enums.limit}" gets dynamically substituted just before matching the pattern to the AST. So, it's going to look for SrcRange=".Src." during pattern matching. In the AST, it'll look for SrcRange=".........Src............." for pattern matching. If it matches that, then the pattern can continue matching the stuff after the SrcRange attribute, otherwise the pattern doesn't match, and nothing gets generated. So, one thing to watch is case, "Src" != "src". Otherwise, set limit="[Ss]rc". These are all C# Regex expressions.
On 1/17/2019 9:53 AM, Darren Schroeder wrote:
ok, i have a new dotnet core console app created. I've changed my csproj to look like this.
|<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>netcoreapp2.2</TargetFramework> </PropertyGroup> <ItemGroup> <PackageReference Include="piggy" Version="1.0.5"> <PrivateAssets>all</PrivateAssets> <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets> </PackageReference> </ItemGroup> <ItemGroup> <Piggy Update="lep.pig"> <ClangSourceFile>include.c</ClangSourceFile> <AstOutputFile>ast.txt</AstOutputFile> <ClangOptions>"ID:/Src/GitHub/Leptonica/src" "ID:/Src/GitHub/Leptonica/build/src"</ClangOptions> </Piggy> </ItemGroup> </Project> |
I have added include.c and lep.pig to the solution.
I build and it waits for a while and then just says failed. I have validated that Piggy1.0.5 is part of the solution via Manage Nuget Packages in Visual Studio.
Do the include.c and the lep.pig need to be in a particular directory? I'm guessing it can't find them.
My detailed build log says this.
|1>Target PiggyCompile: 1> Building target "PiggyCompile" completely. 1> Output file "obj\Debug\netcoreapp2.2\LeptPig.csproj.PiggyGeneratedCodeFileListAbsolute.txt" does not exist. 1> Using "PiggyClassGenerationTask" task from assembly "C:\Users\myusername.nuget\packages\piggy\1.0.5\build..\lib\netstandard2.0\Piggy.dll". 1> Task "PiggyClassGenerationTask" 1> Done executing task "PiggyClassGenerationTask" -- FAILED. |
So, I'm not sure what is going on.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaby76/Piggy/issues/6#issuecomment-455199505, or mute the thread https://github.com/notifications/unsubscribe-auth/AK3KCycyh9XDFNhhzKQx-_0D28fLSj52ks5vEI50gaJpZM4ZpzgG.
OK, just make sure you see FunctionDecl, RecordDecl, TypedefDecl nodes in the AST for the things you want to wrap. -Ken
On 1/17/2019 9:55 AM, Darren Schroeder wrote:
I just noticed that I do have an ast.txt file that is 6,298 kb. So, that part seems to be working.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaby76/Piggy/issues/6#issuecomment-455200228, or mute the thread https://github.com/notifications/unsubscribe-auth/AK3KCxG8BLZjiFbobd55n3Oq08NpDGeCks5vEI7sgaJpZM4ZpzgG.
The nodes look good in the ast.txt file.
Thanks for the explanation. I understand how the included pig files pick up on the limit variable in the local lep.pig. I also am following how it gets substituted and used for matching in the ast.txt. What I'm not following is why a .cs file isn't generated.
As an example, here is one line from my ast.txt file.
( FunctionDecl Pointer="0x21a9aff6f90" SrcRange="D:\Src\GitHub\Leptonica\src\allheaders.h:365:10, D:\Src\GitHub\Leptonica\src\allheaders.h:365:150" SrcLoc="D:\Src\GitHub\Leptonica\src\allheaders.h:365:24" Name="boxaSmoothSequenceMedian" Type="BOXA *(BOXA *, l_int32, l_int32, l_int32, l_int32, l_int32)" Attrs="extern"
I see that it's going to search SrcRange="D:\Src\GitHub\Leptonica\src\allheaders.h for the pattern from the line limit = "[Ss]rc"; in my local lep.pig.
[Ss]rc has multiple hits on that line. Do I need to have a better regex that excludes the D:\Src\GitHub\Leptonica\ part of that string? I guess I'm not clear on what the purpose of this src pattern matching is all about? Are you just trying to find the allheaders.txt file, in my example?
Yeah! I got a lep.pig.cs by taking the ast.txt file and running it through PiggyTool. Looking at it now.
Yeah, "Src" is just a pattern that it tries to find in "|D:\Src\GitHub\Leptonica\src\allheaders.h". Should work. the AST node looks ok.|
|Hmm. BTW, I'm just now fixing capture of the tool output for the build. ||Try |running the piggy tool by hand, something like this....
dotnet "C:\Users\Kenne.nuget\packages\piggy\1.0.5\build\PiggyTool.dll" -a "ast.txt" -s "lep.pig" -t "C:\Users\Kenne.nuget\packages\piggy\1.0.5\Templates" -o "obj\Debug\netcoreapp2.2\lep.pig.cs"
or even without the -o "..."-----it'll go to stdout.
Ken
On 1/17/2019 10:34 AM, Darren Schroeder wrote:
The nodes look good in the ast.txt file.
Thanks for the explanation. I understand how the included pig files pick up on the limit variable in the local lep.pig. I also am following how it gets substituted and used for matching in the ast.txt. What I'm not following is why a .cs file isn't generated.
As an example, here is one line from my ast.txt file. |( FunctionDecl Pointer="0x21a9aff6f90" SrcRange="D:\Src\GitHub\Leptonica\src\allheaders.h:365:10, D:\Src\GitHub\Leptonica\src\allheaders.h:365:150" SrcLoc="D:\Src\GitHub\Leptonica\src\allheaders.h:365:24" Name="boxaSmoothSequenceMedian" Type="BOXA *(BOXA *, l_int32, l_int32, l_int32, l_int32, l_int32)" Attrs="extern"|
I see that it's going to search |SrcRange="D:\Src\GitHub\Leptonica\src\allheaders.h| for the pattern from the line |limit = "[Ss]rc";| in my local lep.pig.
|[Ss]rc| has multiple hits on that line. Do I need to have a better regex that excludes the |D:\Src\GitHub\Leptonica| part of that string? I guess I'm not clear on what the purpose of this src pattern matching is all about? Are you just trying to find the allheaders.txt file, in my example?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaby76/Piggy/issues/6#issuecomment-455215070, or mute the thread https://github.com/notifications/unsubscribe-auth/AK3KCzr9iQTYx69EhYhApESSJ-UvEnPEks5vEJgigaJpZM4ZpzgG.
LOL. I guess Piggy doesn't yet understand parameters named with data type names.
public static extern IntPtr sarrayCreateWordsFromString(string string);
Eeks. Yeah, it should rename the parameter name. Will add an "@" in front if it's a base type name when I move all that mapping code from PiggyRuntime into the templates. You can do that manually for now, then continue seeing what else is bad. Thanks. -Ken
On 1/17/2019 11:01 AM, Darren Schroeder wrote:
LOL. I guess Piggy doesn't yet understand parameters named with data type names. |public static extern IntPtr sarrayCreateWordsFromString(string string);|
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaby76/Piggy/issues/6#issuecomment-455226048, or mute the thread https://github.com/notifications/unsubscribe-auth/AK3KC7p_UzQzDqS53RJTWFMRbjnSsgVrks5vEJ5YgaJpZM4ZpzgG.
I have a project mostly working that uses the .cs generated from the command line with piggy. Changes:
- Change DllImport with the name of a real dll.
public const string LeptDll = "leptonica-1.77.0d.dll";
[DllImport(LeptDll, CallingConvention = CallingConvention.Cdecl, EntryPoint = "pixCleanBackgroundToWhite")]
-
replace all occurances of
string stringwithstring @string -
add this file to the solution as FILE.cs (not sure it all works but it compiles) FILE.txt
Programs.cs loads a file.
class Program
{
public const string feyn = @"D:\Src\GitHub\leptonica.net\images\feyn.tif";
static void Main(string[] args)
{
var p = Functions.pixRead(feyn);
}
}