WhisperKit icon indicating copy to clipboard operation
WhisperKit copied to clipboard

Index out of range error in TextDecoder

Open cgfarmer4 opened this issue 3 months ago • 5 comments

Occasionally Im seeing an index out of range crash on the segmentLogProbs[index] after a long period of silence. https://github.com/argmaxinc/WhisperKit/blob/main/Sources/WhisperKit/Core/TextDecoder.swift#L518-L521

Swift/ContiguousArrayBuffer.swift:600: Fatal error: Index out of range

Two ways I could see guarding against this:

  1. Use swift zip
  2. Check the index against segmentLogProbs count.
for (token, logProb) in zip(segmentTokens, segmentLogProbs) {
    tokenProbs.append([token: logProb])
}

for (index, token) in segmentTokens.enumerated() {
  if index < segmentLogProbs.count {
      tokenProbs.append([token: segmentLogProbs[index]])
  }
}

Happy to PR either one but unsure if Im missing a reason for this being as is.

cgfarmer4 avatar Mar 08 '24 04:03 cgfarmer4