pytorch-transformer
pytorch-transformer copied to clipboard
hkproj
Attention is all you need implementation
### What I have done? - I have cloned the project in my Macbook Pro and using Jupyter Notebook, I have run the file https://github.com/hkproj/pytorch-transformer/blob/main/Local_Train.ipynb How to solve the code...
According to the formula norm = (x - mean) / sqrt(var + eps) not (x - mean)/(std + eps) sqrt(var + eps) == sqrt(std**2 + eps) != (std + eps)...
If input sentence is (A, B, C, D, PAD). In this implementation, encoder mask is [[[FALSE, FALSE, FALSE, TRUE]]] But the encoder attention is [ [AA, AB, AC, AD], [BA,...
Hi @hkproj , I found an issue with the latest_weights_file_path() function in config.py. The original code uses weights_files.sort() to sort the files and selects the last one as the latest...
Hi @hkproj, I found an issue with the BLEU score calculation in train.py. The torchmetrics.BLEUScore() function expects a list of reference sentences but receives a single sentence instead. Here is...
class Encoder(nn.Module): def __init__(self, features: int, layers: nn.ModuleList) -> None: super().__init__() self.layers = layers self.norm = LayerNormalization(features) def forward(self, x, mask): for layer in self.layers: x = layer(x, mask) return...
Even after training for 30 epochs with batch size 32, lr of 1e-4, I the predicted results are very poor. What can be done? ``` -------------------------------------------------------------------------------- SOURCE: Karenin was arguing...
Colab training Fix: Clean up unused import and update local configuration for Colab training 1 . Removing unused dependencies. 3. Updating local file paths in the configuration for smooth execution...
