Reinserted language boxes aren't quite long enough?
At some point, the behaviour of automatic language boxes seems to have changed. Open a new PHP&Python file, and start typing function -- immediately a Python language box appears. Then type f (i.e. space then "f") -- the box will disappear. Delete f and the box will reappear but it will no longer encompass the space character, so whatever you now type is outside of the box. That doesn't seem entirely intuitive, although I'm not sure what the perfect fix might be.
Yes, I believe this is due to the shrinking/expanding feature of automatic language boxes (which we didn't have before so this wasn't an issue before). The problem are whitespaces within language boxes. So whenever we shrink a box, we shrink past any whitespaces within the box, because the other way is also uninuitiv and leaves loads of language boxes with trailing whitespaces behind. The problem is that the cursor stays where it is, and thus we move outside of the box. We could just move the cursor along with the language box whenever we shrink, but that may have other unforeseen consequences and confuse the user as well or even more.
The other alternative might be to resize up to the cursor, if there's only whitespace between the "initially reinserted" box and the cursor?
Hm, that might work. I will give this a go and see how it holds up in practice.
Turns out this is a bit more complicated. It actually has nothing to do with shrinking (I don't know why I thought it did). Since the box gets entirely removed again after typing f, we have no idea it was ever there when we remove f again. So when we remove f we just insert a completely new box again and the default is to exclude trailing whitespace when doing so. I'm not sure if there is an easy way to check this. We need to check if we had a box there before and whether that box had trailing whitespace and if that is the case any new box inserted will also be allowed to have trailing whitespace.
I think my suggestion is a hack that means we don't need to remember the previous box: we would just say "if my current change inserts a box, and there's whitespace inbetween its end and the cursor, just extend the box up to the cursor".
Yes, we used to do something like that, but that leaves us with loads of boxes with trailing whitespaces and we conciously made the decision to exclude them. Because most of the time we want the whitespace in the outer languages. This is a rare example where we don't. But I'm not sure how to differentiate it from the others.
Maybe when you move the cursor out of a box, we trim the box (in the sense that we move trailing whitespace out of the box) automatically?
Hmm, that might work. I reason about that for a bit.