JackCaoG
JackCaoG
Issue was coming from that when we uploading a `at::Scalar` to the device, the current codegen upload does not take a scalar type, hence we always upload with the default...
@wonjoolee95 I think this one is ready for reivew
Hi @ibeltagy , I am working on the lowering part but it is a bit tricky. You will see the pr linked in this issue when it is ready 😄.
HI @ibeltagy I am not sure if 1 hour is too long, it really depends on your model size. Did you remember how much time it takes prior to the...
Hi @ibeltagy 5 minutes to 1 hour seems a big jump. One possibility is that `unfold` was not lowered prior to this change and it is a pretty complex lowering...
Do you mind trying out the idea of splitting the tensor before unfold and concat the result afterward? something like ``` >>> torch.arange(12).reshape([2,2,3]).unfold(1, 2, 1) tensor([[[[ 0, 3], [ 1,...
I pushed a new change to the unfold pr, the peak memory usage should be reduced to 1/3 when step > 3.
Hi @ibeltagy , glad to hear that you get the `unfold` working. Let's keep this thread about `unfold` and open a new issue for the performance optimization😄 .
Fore sure, we still want `unfold` to be lowered in a way that is usable for you. We are a small team we have to pick tasks carefully, since this...
@JunwenBai I believe it is [this](https://github.com/ibeltagy/transformers/blob/xla/src/transformers/modeling_longformer.py#L142) function