Support Unity

[DanMcLaughlin]

Unity3D (http://unity3d.com) uses Mono for the C# engine, but Antlr doesn't yet support that runtime.

[DanMcLaughlin]

Unity regenerates the project files at will. When it does obviously all the custom project settings are lost.

I can write a .cs file that when added to Unity (must go into a folder below "Assets", e.g. "Assets/VSAntlrIntegration/VSHooks.cs") will have hooks for me to write to the project file whenever Unity generates it.

The hook template is here http://unityvs.com/documentation/api/project-file-generation/

so ...

• How to have the NuGet package put a file here (%PROJ_ROOT%/Assets/VSAntlrIntegration/VSHooks.cs)?
• What to write in that template such that project file will have the hooks to, e.g. auto generate the lexer/parser from the grammar? *I also see that the "packages.config" file is getting lost from the project file, but I'm not sure that's necessary. *All files to be recognized by Unity must go into Assets as mentioned, a nice trick might be to have a setting for where generated cs files go (perhaps this already exists). That way you could keep the grammars outside of Assets (Unity doesn't care about them) but the grammars do get put into Assets.
• Likewise the Runtime dll must go into Assets for Unity to use it

Note: Unity uses it's own compiler (Mono based I believe), so compiling in Visual Studio doesn't matter one way or another. Further it doesn't contain all the language features supported in VS.

I think that'll provide proper synchronization between the two worlds.

[sharwell]

It's pretty straightforward to include a code file in a NuGet package which gets added to the project. This file would add the necessary references to the project file pointing to the ANTLR 4 code generator. Note that updating the version of ANTLR 4 used in a Unity project would request permission to overwrite this file because it exists in a location that users can see and modify.

I'm hoping we can find a way to include this support in the existing Antlr4 package on NuGet, but if we can't figure out the correct framework-specific folder then we may have to fall back to creating a Unity-specific package in addition to the existing one.

[DanMcLaughlin]

I haven't had a chance yet to experiment to find the name yet. If it helps, I get

".NETFramework,Version=v3.5,Profile=Unity Subset v3.5"

found via the pointers from http://blog.stephencleary.com/2012/05/framework-profiles-in-net.html

[sharwell]

If you're feeling bored (:wink:), the biggest thing you could do aside from figuring out the folder name is to help write the VSHooks file. The necessary parts to include in the project file are the following:

1. An <Import> line at the top of the file for the .props file
2. A reference to the runtime assembly (this is not really custom so it might be handled for you)
3. An <Import> element at the bottom of the file for the .targets file

[DanMcLaughlin]

Well, it's easier than working on the C++ Antlr port so yeah I'll do it!

[DanMcLaughlin]

OK I've written the hooks file to do this. Tricky, or not so tricky after you figure it out. Anyhow I also notice that the packages.config file gets added as part of the ANTLR package but then gets wiped from the Unity generated project file. Important?

EDIT: Well, whatever, I added it. Now all that needs to be done is the part we're both avoiding which is figuring out the namespace :) Rest of it looks to be working fine at any rate.

On testing however the parser/lexer aren't regenerating on changes to the grammar. I'm not seeing anything else missing from the project files. On a build at least it does generate them in obj/Debug (with errors), but the two files in the same directory as the grammar aren't updating, and those are the two that Unity will see.

[sharwell]

That's great news, even if we haven't figured out 100% of the issues. :+1:

Anyhow I also notice that the packages.config file gets added as part of the ANTLR package but then gets wiped from the Unity generated project file.

I assume you are stating that the following line gets removed from the .csproj file.

<None Include="packages.config" />

Now all that needs to be done is the part we're both avoiding which is figuring out the namespace

I didn't follow this. If it's a problem, can you provide an example (or just clarify that it's not a problem)?

On testing however the parser/lexer aren't regenerating on changes to the grammar.

I expected this at least initially. I'll try to find a workaround.

but the two files in the same directory as the grammar aren't updating, and those are the two that Unity will see.

Which two files? The generated code should only ever appear in the obj/$(Configuration) folder. If they appeared alongside the grammar we'll have to figure out why they were placed there.

[DanMcLaughlin]

I didn't follow this. If it's a problem, can you provide an example (or just clarify that it's not a problem)?

I'm talking about the packaging namespace so that the NuGet package will install into Unity without complaints (I codged it together by pulling out the 3.5 package and putting it in the root of the zip file) (e.g. the comment above "I haven't had a chance yet to experiment to find the name yet. If it helps, I get ".NETFramework,Version=v3.5,Profile=Unity Subset v3.5" ... found via the pointers from http://blog.stephencleary.com/2012/05/framework-profiles-in-net.html")

Which two files? The generated code should only ever appear in the obj/$(Configuration) folder. If they appeared alongside the grammar we'll have to figure out why they were placed there.

When I added a grammar it auto created a lexar and parser in the same directory, but they never update after that. Unity seemingly ignores everything in the obj directory (obviously knows that's Visual Studio foo and doesn't touch it). In fact those directories don't even show up in the Unity hierarchy. For Unity to recognize and compile the files they need to show up in the "Assets" directory, e.g. next to the grammar file (which is shows but ignores otherwise, assuming it's a metadata file probably).

Edit: the Studio compiles are failing on the generated lexar/parser for some reason. May be important later, but its the Unity compile that matters.

[DanMcLaughlin]

On re-reading I think I should clarify. It appears that the obj/$(Configuration) files are getting regenerated properly, and the fact that a lexer/parser at the same level in the hierarchy is initially generated isn't a problem. The problem actually is that those files (the lexer/parser not in obj) aren't then kept in sync.

So I think that the existing code which generates files in obj/$(Configuration) can be left alone as it won't do any harm. If you can figure out how to have the versions that get dropped next to the grammar stay in sync then I think we're good (maybe as simple as adding a rule to generate those for this configuration). Assuming it all compiles in Unity just fine. I can try copying the obj/... generated lexer/parser up to see what Unity thinks of them.

Edit: FYI if I add a grammar Java.g4 it will add two files Java.g4.lexer.cs and Java.g4.parser.cs in the same directory. In the obj/$(Configuration) directory it will create a bigger set, e.g. JavaBaseListener.cs, JavaBaseVisitor.cs, etc. So rather those files need to go up to next to the grammar, or in a directory below that.

At this point I need your help so if you could look into that I'd appreciate it.

[DanMcLaughlin]

EDIT: ~~Another~~ ~~issue.~~ ~~Copying~~ ...

Wups, turned out to be some custom Java code you put in the Grammar which was injected into the final, I thought that was a no-no :) :)

Now I'm getting what looks like an assembly error on compilation

_compute_class_bitmap: Invalid type 13 for field Antlr4.Runtime.Recognizer`2[Symbol,ATNInterpreter]:interp

Somebody else has noticed this, looks like an issue with Mono using the Antlr dll's

http://stackoverflow.com/questions/22375476/antlr4-with-unity3d

[DanMcLaughlin]

Giving up on this idea for now.

Unity is using an older version of Mono (2.0.50727.1433) because of licensing issues. This bug (compute_class_bitmap) has been fixed, but not in this ancient version. Meanwhile Unity has created a IL to C++ converter to deal with this issue, but that will only support static .NET features and no dynamic such as reflection (which I'm pretty sure is used by antrlcs?) And at any rate that translation presently only supports iOS 64 bit runtime.

A lesson I learn again and again - third party tools will screw you, every time. Always run on the bare metal as much as you can.

[sharwell]

Hi Dan,

Thanks for looking into this as much as you did. I've been swamped, but I really do want to address this and I'm certain your work and commentary will prove instrumental regardless of the outcome (whether it is a successful implementation or determination that it can't be implemented). :+1:

[DanMcLaughlin]

I'll post the VS project file hooks code here for reference but I think it's hopeless. With the Xamarin/Mono licensing situation its unlikely Unity will ever update, and now their stated policy is to move to their translation tool (e.g. "run away from Mono"). It's possible that the tool could successfully translate the Antlr dll to C++ but seems unlikely as they state they won't support runtime features (like reflection).

The one possibility is compiling the ANTLR source directly within Unity. For jollies I might try that but I suspect it will give the same problem.

[DanMcLaughlin]

For the record, here's the bug (fixed in 2010), appears to be an issue with two parameter templates

https://bugzilla.novell.com/show_bug.cgi?id=594942

Here's the project file hooks

---

using System;
using System.IO;
using System.Linq;
using System.Text;
using System.Xml.Linq;

using UnityEngine;
using UnityEditor;

using SyntaxTree.VisualStudio.Unity.Bridge;

[InitializeOnLoad]
public class ProjectFileHook
{
    // necessary for XLinq to save the xml project file in utf8
    class Utf8StringWriter : StringWriter
    {
        public override Encoding Encoding
        {
            get { return Encoding.UTF8; }
        }
    }

    static ProjectFileHook()
    {

        ProjectFilesGenerator.ProjectFileGeneration += (string name, string content) =>
        {
            var document = XDocument.Parse(content);
            document.Root.Add(new XComment("Antlr Hooks Added"));

            string path = SyntaxTree.VisualStudio.Unity.Bridge.Project.Directory(null);
            document.Root.Add(new XComment("Path: " + path));

            if (!name.Contains("CSharp.Editor.csproj"))
            {
                XNamespace ns = document.Root.GetDefaultNamespace();

                // .props file
                XElement importElement = new XElement(ns + "Import", new XAttribute("Project", "packages\\Antlr4.4.3.0\\build\\Antlr4.props"), new XAttribute("Condition", "Exists(\'packages\\Antlr4.4.3.0\\build\\Antlr4.props\')"));
                document.Root.AddFirst(importElement);

                // g4 files
                XElement packagesElement = new XElement(ns + "ItemGroup", "");
                packagesElement.Add(new XElement(ns + "None", new XAttribute("Include","")));
                document.Root.Add(packagesElement);

                // packages.config
                XElement g4Element = new XElement(ns + "ItemGroup", "");
                g4Element.Add(new XElement(ns + "None", new XAttribute("Include", "packages.config")));
                document.Root.Add(g4Element);

                XElement targetsElement = new XElement(ns + "Import", new XAttribute("Project", "packages\\Antlr4.4.3.0\\build\\Antlr4.targets"), new XAttribute("Condition", "Exists(\'packages\\Antlr4.4.3.0\\build\\Antlr4.targets\')"));
                document.Root.Add(targetsElement);

            }
            var str = new Utf8StringWriter();
            document.Save(str);

            return str.ToString();
        };
    }
}

[MSylvia]

I just hit the same error. I created a minimal project that shows the error based on the bug link that @DanMcLaughlin referenced and submitted a bug report to Unity with all of the information.

Unity bug #816232 : https://fogbugz.unity3d.com/default.asp?816232_jkdd76ddpsfuvpbg

[MSylvia]

Just got an email from unity saying the issue @DanMcLaughlin mentioned is resolved in an upcoming release.

Thanks for submitting this bug report. We have corrected the issue, and the fix should be available in the 5.3.6p4 and the 5.4.0p3 patch releases. Please let me know if you have any questions. Thanks again!

Unity bug #816232 : https://fogbugz.unity3d.com/default.asp?816232_jkdd76ddpsfuvpbg