[Bug]: LLMs have difficulties generating new line characters "\n", which is extremely obvious while generating code blocks in markdown

[nath1295]

Your current environment

PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect 
CMake version: version 3.29.3
Libc version: glibc-2.35
Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA RTX 5000 Ada Generation
GPU 1: NVIDIA RTX 5000 Ada Generation

Nvidia driver version: 550.90.07
cuDNN version: Could not collect 
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             64
On-line CPU(s) list:                0-63
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 9354P 32-Core Processor
CPU family:                         25
Model:                              17
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          1
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        3799.0720
CPU min MHz:                        1500.0000
BogoMIPS:                           6490.85
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualisation:                     AMD-V
L1d cache:                          1 MiB (32 instances)
L1i cache:                          1 MiB (32 instances)
L2 cache:                           32 MiB (32 instances)
L3 cache:                           256 MiB (8 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-63
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] onnxruntime==1.18.0
[pip3] torch==2.3.0
[pip3] triton==2.3.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] torch                     2.3.0                    pypi_0    pypi
[conda] triton                    2.3.0                    pypi_0    pypiROCM Version: Could not collect 
Aphrodite Version: 0.5.3
Aphrodite Build Flags:
CUDA Archs: Not Set; ROCm: Disabled

🐛 Describe the bug

While serving the llm, it seems that there are something that stops the LLM from generating new line characters. It happens to different quantisation methods. I have tried exl2, gguf, and load in smooth with different models. All of them seems to have the same problem, which does not happen while using other inferencing engine like llama.cpp for the same model files.

A few examples I have tried:

 aphrodite run LoneStriker/Nous-Hermes-2-Mixtral-8x7B-DPO-4.0bpw-h6-exl2 \
-q exl2 --port 5002 \
--kv-cache-dtype fp8 \
--served-model-name Nous-Hermes-2-Mixtral-8x7B-DPO --max-model-len 32768 -gmu 0.95 --download-dir /my_dir/Nous-Hermes-2-Mixtral-8x7B-DPO-4.0bpw-h6-exl2

aphrodite run NousResearch/Hermes-2-Pro-Mistral-7B \
--load-in-smooth --port 5002 \
--served-model-name Hermes-2-Pro-Mistral-7B --max-model-len 32768 -gmu 0.8 --download-dir /my_dir/Hermes-2-Pro-Mistral-7B

aphrodite run /my_dir/models--bartowski--Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q8_0.gguf \
--tokenizer thesven/Mistral-7B-Instruct-v0.3-GPTQ \
-q gguf --port 5002 --served-model-name Mistral-7B-Instruct-v0.3 --max-model-len 32768 -gmu 0.6

One example output:

'To accomplish this task, we will use the MNIST dataset, which is a popular handwritten digit classification dataset. We\'ll use PyTorch and torchvision to download and load the data, and then train a simple neural network to classify the images.\n\nHere\'s a Python script that demonstrates this process:\n\n```python\nimport torch\nfrom torch import nn, optim\nfrom torchvision import datasets, transforms, models\nimport torchvision.transforms as transforms_tf\nfrom matplotlib import pyplot as plt  # for displaying image samples later on (optional)\nimport numpy as np  # for converting array data types (optional)\n# To avoid any issues with GPU memory limits during training in Google Colab environment. This line ensures that only one GPU is used if available. In your local machine setup you can remove this line or uncomment `device = \'cuda\'` based on your setup.  # noqa: D401  # pylint: disable=no-member   # noqa: F821; in scoped function; pylint: disable=protected-access    # noqa: F821; in scoped function; pylint: disable=redefined-outer-name    # noqa: F821; in scoped function; pylint: disable=redefined-outer-name   # noqa: F821; in scoped function; pylint: disable=redefined-outer-name     device = \'cpu\' if not torch.cuda.is_available() else \'cuda\'  ## No need to change this line unless you have specific requirements ## Here are some suggested changes for better understanding/usefulness of the code commenting out unnecessary parts related to google colab environment ##################################   device = \'cuda\' iftorch.__version__ >= "1" else "cpu" if not isinstance(device, str) or (isinstance(device, str) and device != "cpu" and device != "cuda"): raise ValueError("Invalid Device Type") else None ##################################   def train_net():     """ Download MNIST Dataset"""     transform = transforms_tf .Compose([transforms_tf .ToTensor(),                              transforms_tf .Normalize((0.5,(0)),(0.5,(0)))])     trainingDataMNIST = datasets .MNIST(\'~/.pytorch/MNIST_data/\',train=True ,download=True ,transform=transform ) ## Specify path where mnist should be downloaded / stored - it needs to be outside of /tmp bacause it seems google colab clears that after every run    ## If running locally just uncomment these lines - ~/.pytorch/MNIST_data/ would refer to /home/$USERNAME/.pytorch/MNIST_data/# uncomment these lines for local use###trainingDataMNIST = datasets .MNIST(\'.\' ,train=True ,download=True ,transform=(lambda x :x)) ###############32 classes instead of 10 because of test set split changed from validation set -> test set so we don\'t have labeles now for classes we need them !!!!## trainingDataLabels = trainingData[\'target\'] print(\'\\n Train Dataset Size : \', len(trainingDataMN Ist[\'data\'])) print(\'\\n Label Size : \', len(trainingDataLabels)) class Net(nn .Module):        def __init__(self):            super().__init__()            self._initialize()        def _initialize(self):             self._layers = nn .Sequential(\\             nn .Conv2d (3 ,6 ,5 ),\\             nn .ReLU(),\\             nn .MaxPool2d ((2 ,2)),\\             \\               )*3 \\           +nn.\\Linear((6*7)*6,\\t     3),       )         self._layers +=nn.\\Dropout()         self._layers +=nn.\\Softmax(\\t      size=(3))         self._output=\\t      "" output doesnt really matter here since were just testing functionality """ Now let us implement some functions like forward pass etc ... """          def forward(self,\\x):               x =\\               self(_layers)(x)\\              return x          def trainmodel():              model =\\           Net().to(device)              lossfunction =\\           nn.\\CrossEntropyLoss()                                       optimizer =\\           optim.\\SGD(\\model\'\\ parameters(), lr=\\                           5e -\\              4,\\weight\\_decay=\\                       9e-\\              5,\\momentum=\\               0,\\verbose=(False))              total\\_step =\\            len(\\train\\_dataset\\/batch\\_size)\\         '