openvino icon indicating copy to clipboard operation
openvino copied to clipboard

[GPU] Add Validate pass call after IncreasePositionIdsPrecision to ensure proper data type propagation

Open sshlyapn opened this issue 1 year ago • 0 comments

Details:

This patch adds Validate pass call after IncreasePositionIdsPrecision to ensure proper data type propagation

With this change the accuracy of llama-3-8b INT8 (and other LLMs probably) can be restored to expected level Before:

| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6030|±  |N/A   |
|        |       |none  |     0|byte_perplexity|↓  |1.5189|±  |N/A   |
|        |       |none  |     0|word_perplexity|↓  |9.3472|±  |N/A   |

After:

| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.5351|±  |N/A   |
|        |       |none  |     0|byte_perplexity|↓  |1.4490|±  |N/A   |
|        |       |none  |     0|word_perplexity|↓  |7.2664|±  |N/A   |

Tickets:

sshlyapn avatar Aug 29 '24 08:08 sshlyapn