LAVA support for ConvT layers in LAVA-DL training

[alexggener]

User story

As a user, I want to be able to test in LAVA networks that contains ConvT layers supported by LAVA-DL training.

Conditions of satisfaction

• Be able to export SNNs containing ConvT layers in LAVA-DL in h5py
• Be able to perform inference of ConvT layers in LAVA and deploy them in loihi2

[mathisrichter]

Thanks, @alexggener for the feature request. We will review this item and may get back to you with questions.

[bamsumit]

Hi @alexggener would you be interested in implementing ConvT process in lava since you already helped on the implementation of ConvT blocks on lava-dl? I can point you toward how it can be done (especially in a way that it can be easily enabled on loihi).

[alexggener]

Hi Sumit,

That sounds great. We could work on this at the beginning of Q2 if that suits for you.

Alex.

From: bamsumit @.> Sent: 09 February 2023 15:43 To: lava-nc/lava @.> Cc: Garcia Gener, Alejandro Export License Required - US Collins @.>; Mention @.> Subject: [External] Re: [lava-nc/lava] LAVA support for ConvT layers in LAVA-DL training (Issue lava-nc/lava#619)

Hi @alexggenerhttps://urldefense.com/v3/__https:/github.com/alexggener__;!!MvWE!EYr8kNTF3hjAx1nqL8Licd7bEwkYNkqkNpbbRklSutMQxBllR2xuvOAZEjaI_c0ZuvYsiF7JeGlN786Gddhqn0PpJtNjspDQNg$ would you be interested in implementing ConvT process in lava since you already helped on the implementation of ConvT blocks on lava-dl? I can point you toward how it can be done (especially in a way that it can be easily enabled on loihi).

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/lava-nc/lava/issues/611*issuecomment-1424402811__;Iw!!MvWE!EYr8kNTF3hjAx1nqL8Licd7bEwkYNkqkNpbbRklSutMQxBllR2xuvOAZEjaI_c0ZuvYsiF7JeGlN786Gddhqn0PpJtOQPNYm6g$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AW2SAYSA6SPP4H7RR6THSODWWUGBVANCNFSM6AAAAAAUVLXNYE__;!!MvWE!EYr8kNTF3hjAx1nqL8Licd7bEwkYNkqkNpbbRklSutMQxBllR2xuvOAZEjaI_c0ZuvYsiF7JeGlN786Gddhqn0PpJtNKNW603Q$. You are receiving this because you were mentioned.Message ID: @.@.>>

[bamsumit]

Hi @alexggener great. Early Q2 sounds fine. Drop me an email or message.

[alexggener]

Hi Sumit,

Bringing back to life this topic. Could you please point us to some details to introduce ConvT blocks into LAVA inference?

Thanks, Alex.

From: bamsumit @.> Sent: 13 February 2023 15:41 To: lava-nc/lava @.> Cc: Garcia Gener, Alejandro Export License Required - US Collins @.>; Mention @.> Subject: [External] Re: [lava-nc/lava] LAVA support for ConvT layers in LAVA-DL training (Issue #619)

Hi @alexggenerhttps://urldefense.com/v3/__https:/github.com/alexggener__;!!MvWE!BcooZdW4HQAVmdh7Y5cLdQ5CtPwYlA6RoHNj8i7biPexEjpxeRnKjsoBE2EAQyKaUevkWLiaP-D3UpEgoOt3VGRc4WPt20XIcg$ great. Early Q2 sounds fine. Drop me an email or message.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/lava-nc/lava/issues/619*issuecomment-1428160956__;Iw!!MvWE!BcooZdW4HQAVmdh7Y5cLdQ5CtPwYlA6RoHNj8i7biPexEjpxeRnKjsoBE2EAQyKaUevkWLiaP-D3UpEgoOt3VGRc4WP9o1ZEnA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AW2SAYVQZOHZNFEINLKDWUDWXJIYXANCNFSM6AAAAAAU2CRMHY__;!!MvWE!BcooZdW4HQAVmdh7Y5cLdQ5CtPwYlA6RoHNj8i7biPexEjpxeRnKjsoBE2EAQyKaUevkWLiaP-D3UpEgoOt3VGRc4WNV6-16wQ$. You are receiving this because you were mentioned.Message ID: @.@.>>

[bamsumit]

Hi @alexggener, great! Here are the coarse steps:

1. create a ConvT process (class ConvT in src/lava/proc/convT/process.py)
2. create a ConvT PyProcModel. (src/lava/proc/convT/models.py)
3. add ConvT utilities to src/lava/proc/convT/utils.py

You can follow the structure for Conv process for these.

I am guessing you are more interested in getting it running on Loihi 2. Keeping that in mind, the best way to approach this problem is to work on ConvT to Conv translator in step 3. ConvT is the same as convolution with 180 rotation of the kernel and appropriate adjustment to padding, dilation, and so on. Once you do this conversion, we can reuse all the machinery for Conv to calculate ConvT and support it easily.

[bamsumit]

Here is an example conversion for basic convolution with stride 1:

import numpy as np
import torch
import torch.nn.functional as F

for i in range(10):
    kernel_x = np.random.randint(7) + 1
    kernel_y = np.random.randint(7) + 1
    ch_in = np.random.randint(10) + 1
    ch_out = np.random.randint(10) + 1
    x_in = np.random.randint(100) + 1
    y_in = np.random.randint(100) + 1
    # stride = np.random.randint(5) + 1
    # padding = np.random.randint(5)
    # dilation = np.random.randint(5) + 1
    stride = 1
    padding = 0
    dilation = 1
    kernel = torch.rand([ch_in, ch_out, kernel_y, kernel_x])
    convT_input = torch.rand([1, ch_in, y_in, x_in])
    convT_output =  F.conv_transpose2d(convT_input, kernel, stride=stride, padding=padding, dilation=dilation)
    convT = F.conv2d(convT_input,
                     kernel.permute([1, 0, 2, 3]).flip([2, 3]),
                     padding=(kernel_y - 1, kernel_x - 1),
                     dilation=1)
    print(f'{i=} {torch.norm(convT - convT_output)=}')

Your task is to generalize it for arbitrary stride, padding and dilation and verify it.