Feature request: add support of runtime compiled parts of kernel

[mikhail-khalizev]

Faced the following problem. There is a common kernel logic. And there are additional methods for transformation numbers that are generated dynamically before starting the kernel (as in the example below, which fails).

I propose to add support for executing Action and Func through a mechanism like SpecializedValue.

using ILGPU;
using ILGPU.Algorithms;
using ILGPU.Runtime;
using System;
using System.Linq;
using System.Linq.Expressions;

namespace SimpleKernel
{
    public class Program
    {
        private static Func<int, int> _transform;

        static unsafe void MyKernel(ArrayView<int> dataView)
        {
            if (Grid.GlobalIndex.X == 0)
                dataView[0] = _transform(10);
        }

        static unsafe void Main()
        {
            var arg = Expression.Parameter(typeof(int), "arg");

            // Runtime generated transformation of number.
            _transform = Expression.Lambda<Func<int, int>>(
                Expression.Add(
                    arg,
                    Expression.Constant(1)),
                arg).Compile();


            using var context = new Context();
            
            foreach (var acceleratorId in Accelerator.Accelerators)
            {
                using var accelerator = Accelerator.Create(context, acceleratorId);

                Console.WriteLine($"Performing operations on {accelerator}");
                        
                var kernel = accelerator.LoadStreamKernel<ArrayView<int>>(MyKernel);
                var config = accelerator.ComputeGridStrideLoopExtent(1);

                using (var buffer = accelerator.Allocate<int>(1))
                {
                    buffer.MemSetToZero();

                    kernel(config, buffer);
                    accelerator.Synchronize();
                    var data = buffer.GetAsArray()[0];
                            
                    Console.WriteLine($"Data: {data}");
                }

                Console.WriteLine();
            }
        }
    }
}

[faruknane]

@mikhail-khalizev I am working on a library that allows you to write dynamic functions and to use/call them in statically created kernels. In a few days, I will try to share two examples of how you can do what you wanted to do.

[faruknane]

@mikhail-khalizev I am sharing a demo example. You can edit the code below depending on your needs. You can create Func<int, int> _transform using Lambda Expression. I will also share a second example to show you how to create dynamic methods easily with the ILDynamics library. You can add the library as a reference using nuget . However, this is not a mature library. So, check if things work well for you. If not, you can always open an issue to the github repo.

public static int Transform(int arg)
{
    return 0;
}

public static void MyKernel(Index1 index, ArrayView<int> dataView)
{
    dataView[index] = Transform(10);
}

static unsafe void Main()
{   
    Func<int, int> _transform = a => a + 1;

    var kernelfunc = typeof(Program).GetMethod("MyKernel");

    //we need to convert func<int,int> to a static function
    //func<int,int> lambda functions have an additional parameter for closure at index 0. We have to remove it!
    var transformfunc = Resolver.CopyMethod(_transform.Method, new ParameterRemover(0), new NoFilter());

    //check if static transform function works properly! I suggest doing the checkings always if something goes wrong! Add assert maybe.
    Console.WriteLine(transformfunc.Invoke(null, new object[] { 1 })); //1 + 1 should write 2!

    //Now we need to swap Program.Transform with our transformfunc!

    //lets create a filter for that!
    var swapper = new MethodCallSwapper();
    swapper.AddSwap(typeof(Program).GetMethod("Transform"), transformfunc);

    kernelfunc = Resolver.CopyMethod(kernelfunc, swapper, new NoFilter()); //always add no filter!

    int experimentsize = 10;

    using var context = new Context();

    using (var accelerator = new CudaAccelerator(context))
    {
        Console.WriteLine($"Performing operations on {accelerator}");
        var backend = accelerator.Backend;
        var entryPointDesc = EntryPointDescription.FromImplicitlyGroupedKernel(kernelfunc);
        var compiledKernel = backend.Compile(entryPointDesc, default);

        using (var kernel = accelerator.LoadAutoGroupedKernel(compiledKernel))
        using (var buffer = accelerator.Allocate<int>(experimentsize))
        {
            buffer.MemSetToZero();

            kernel.Launch<Index1>(accelerator.DefaultStream, buffer.Length, buffer.View);
            accelerator.Synchronize();

            var data = buffer.GetAsArray();
            Console.Write($"Data: ");
            foreach (var item in data)
                Console.Write(item + " ");
            Console.WriteLine();
        }

        Console.WriteLine();
    }
}

Output:

[faruknane]

You can also create dynamic methods. It has limited support. Useful for small things. If you encounter a problem or have a feature request, let me know.

Second example:

var kernelfunc = typeof(Program).GetMethod("MyKernel");

Method<int> m = new Method<int>();
var arg = m.NewParam<int>();
m.Return(m.Add(arg, m.Constant(1)));
var transformfunc = m.Create();

//check if static transform function works properly! I suggest doing the checkings always if something goes wrong! Add assert maybe.
Console.WriteLine(m[1]); //1 + 1 should write 2!

//Now we need to swap Program.Transform with our transformfunc!

//lets create a filter for that!
var swapper = new MethodCallSwapper();
swapper.AddSwap(typeof(Program).GetMethod("Transform"), transformfunc);

kernelfunc = Resolver.CopyMethod(kernelfunc, swapper, new NoFilter()); //always add no filter!

[mikhail-khalizev]

Super! Thanks.

[faruknane]

@mikhail-khalizev did it work?

[m4rs-mt]

@mikhail-khalizev @faruknane We currently plan to add (limited) support for lambda functions. This should also cover dynamic code generation via lambda functions. This issue is also related to #463.