getpos returns the wrong result when in visual mode

[sgrj]

If a selection function is called when in visual mode, the function thinks that the cursor is positioned at the beginning of the selection, even if it is at the end.

The following code illustrates the problem.

call textobj#user#plugin('test', {
\      '-': {
\        'select-a': 'at',  '*select-a-function*': 'Select_a_test',
\      }
\    })

function! Select_a_test()
  let startpos = getpos('.')
  echom "Cursor at ".string(startpos)
  echom "Current mode: ".mode()

  return ['V', startpos, startpos]
endfunction

Now edit any file, and in normal mode do ggVGat. Only the first line is selected, and :messages displays Cursor at [0,1,1,0], although the cursor was on the last line. It should only select the last line.

Furthermore, :messages displays Current mode: n, so the function thinks we are in normal mode, although we are in visual line. (I don't know whether that can really be regarded as a bug. But if we knew that we are in visual mode, we could use this information to expand the selection to the outer block, if the visual selection already selects a block.)

[kana]

It is not wrong. Because Select_a_test is not called in Visual mode. When you type at in Visual mode, at will be mapped to an internal stuff to target the text object, and the stuff is something like :<C-u>call Select_a_test()<Return>. So that you are not in Visual mode, you were in Visual mode.

[sgrj]

I see. Thanks for the explanation. So there is no way to know that we were in visual mode and where the cursor was before we typed at?

[kana]

There is no built-in functions to get such information. And the cursor position just before typing at does not matter. What you want to do is do extend the current selection. You can get the last selection by getpos("'<"), getpos("'>") and visualmode(). The missing part is the mode in which a text object was used.

Fortunately, the mode in which a text object was used can be provided by vim-textobj-user. The last problem is to define API to provide this information to custom text objects, in a backward-compatible manner.