Ryan Taylor
Ryan Taylor
Currently, the token sampling for MoD Infini-Former at inference time can result in different length sequences for each observation in the batch. The current workaround is to force the batch...
While running some tests, I noticed several memory inefficiencies that nullify the advantages of the Infini-Transformer. In particular: - The loop over segments needs to be moved out of the...