Caret position affects visual line selection

[hifi]

When selecting whole lines and the caret is somewhere in the middle of some line the selection isn't always a complete line but only a portion of it. Moving the (invisible) caret with h or l usually fixes this but it's annoying.