Missing einops.layers.....Repeat

[boeddeker]

I tried to use repeat for in torch and needed a layer, but strangely it was not there. I know, that I could use einops.layers.torch.Reduce('...', reduction='repeat', ...), but that is confusing to read.

What do you think about adding einops.layers.....Repeat functions to einops?

Here is a toy example, where the last line fails because the function counterpart is missing and the first line is difficult to read:

import einops

t = torch.randn((2, 3))
print(einops.layers.torch.Reduce('a b -> a b c', reduction='repeat', c=4)(t).shape)  # torch.Size([2, 3, 4])
print(einops.repeat(t, 'a b -> a b c', c=4).shape)  # torch.Size([2, 3, 4])
print(einops.layers.torch.Repeat('a b -> a b c', reduction='repeat', c=4)(t).shape)  # AttributeError: module 'einops.layers.torch' has no attribute 'Repeat'

1. Try to collect use-cases

Since einops.repeat exists, I think the same use cases would be valid for a layer. I have one case, where I want to use it in pytorch.

2. Implementation. (optional) Implementing a sketch of your proposal in a code allows detecting possible conflicts and realize possible caveats.

Something like the following in each layers backend file

class Repeat(Reduce):
    def __init__(self, pattern, **axes_lengths):
        super().__init__(pattern, 'repeat', **axes_lengths)

3. Integrity - does it interplay well with existing operations and notation in einops?

I would say yes, it is the counterpart of the function einops.repeat.

4. Readability. This is harder to check, but give it a try. A simple but usable test is to write an exercise sheet with several examples of your extension explained, for others meaning of operation should be guessed. Send this test to a couple of your friends to collect the feedback and see how your notion will be understood. This should also help to improve your ideas to make them more digestible.

I think this is obvious, currently I use einops.layers.torch.Reduce('...', reduction='repeat', ...) and that is confusing.

[arogozhnikov]

Hi Christoph, thank you for following issue template, despite it did not fit.

Please read a relevant discussion here: https://github.com/arogozhnikov/einops/discussions/126

As far as I see, in most cases repeat can be replaced with rearrange + broadcasting (which is also more memory-efficient).

Repeat layer is necessary e.g. if you need to overwrite components of output. Let me know if that's the case for you

[boeddeker]

Yes, I agree, that usually rearrange + broadcasting is the better option. I don't know, how to answer your question, without my concrete case. I try to estimate the speech activity with a time frequency resolution, i.e. is speaker A active at time t and frequency f. But the neuronal network (NN) output is a stack of frequencies and speakers. After the NN I have many options (i.e. hyperparameters), what I can do and all of them expect a time frequency resolution.

With one hyperparameter of my experiment I say, that only the time when a speaker is active should be estimated (i.e. no frequency resolution). While broadcasting can work, it is difficult to implement, because there is so much code, and it is a burden to get the broadcasting right, just for a side experiment.

So, yes, you can always use repeat and broadcasting, but sometimes it is better to just waste some computational time, if the remaining codes get easier to read.

With your argumentation, you could also say, that you never need the function repeat. For computations, there always exists a possibility to do it with rearrange and broadcasting. Sometimes you just have a case, where the computation time / memory consumption doesn't matter and getting rid of several lines of code to distinguish between singleton and non singleton axis it better. At least for the visualization, I have to use repeat, because tensorboard is stupid and cannot properly visualize an image with a height of 1.

[arogozhnikov]

you could also say, that you never need the function repeat

not really, np.repeat and np.tile behavior can't be obtained from other functions. It's just my observation that use-cases for Repeat layer mostly fall into bin "you could broadcast it".

Anyway, I see your point about convenience. I've also recalled that now repeat uses "torch.expand"-like behavior, so it should be quite performant anyway. Will try to incorporate Repeat layer in the next release.

[KarelPeeters]

Having a Repeat layer would also help with #115, since only the layers work with torch.jit.script for now.

[josmople]

Hello, I have this use case where I want to augment an image with multiple different (random) transformations. Having a Repeat layer will be helpful.

torch.nn.Sequential(
    einops.layers.torch.Repeat("B C H W -> (B N) C H W", N=10),
    RandomImageAugmentation(), # B C H W → B C H W, random edit per image in batch
    einops.layers.torch.Rearrange("(B N) C H W -> B N C H W", N=10),  
)