thinc icon indicating copy to clipboard operation
thinc copied to clipboard

`SparseLinear` does not look up weights correctly

Open danieldk opened this issue 3 years ago • 0 comments

Reported in https://github.com/explosion/spaCy/discussions/11379

The weight matrix is allocated as follows:

model.set_param("W", model.ops.alloc((nO * length,), dtype="f"))

But indexed like this:

scores[clas] += weights[idx1 + clas] * value
scores[clas] += weights[idx2 + clas] * value

Where idx1/idx2 is in [0,length) and clas is in [0, nO). So the largest index ever used is (length-1) + (clas-1), not nO * length - 1. Depending on how we look at W, we should either do something like

scores[clas] += weights[(idx1 * nO) + clas] * value

or

scores[clas] += weights[idx1 + (clas * length)] * value

danieldk avatar Sep 01 '22 09:09 danieldk