syntastic

syntastic copied to clipboard

tpope has a new vim plugin out for async jobs. Watch the introductory video.

http://vimeo.com/63116209

https://github.com/tpope/vim-dispatch#readme

@scrooloose is still missing in action, but I'm pretty sure he mentioned vim-dispatch in the past. So yes, we're aware of it.

My personal take is that vim-dispatch is nice and very well written, but it only supports quickfix lists, while syntastic needs loclists. An older patch adding support for async checking (see #370) uses AsyncCommand instead, which can handle both quickfix lists and loclists. I'd still prefer vim-dispatch, mainly because the code is cleaner. But for that to happen we'd have to either persuade @tpope to support loclists, or change syntastic to use quickfix lists (which in turn would make it conflict with :make and friends), or do some ugly dance to bridge quickfix lists with loclists (and still conflict with :make).

Well, we should wait for @scrooloose to get back.

I don't really see these two plugins as a great fit together. Can you imagine a tmux pane opening for a split second every time you write a file? Gross.

If it happens in background, why not? :)

You mean other than two resizes and a potential view port shift?

Ah right, in tmux parlance "panes" are equivalents to Vim's windows. Ok, would that work better if the checkers would be sent to run in a background tmux window (equivalent to a background tab in Vim parlance)?

Dispatch relies on the resize event to trigger the callback. Without that, there's no point in bothering with tmux at all.

What about vimproc? https://github.com/Shougo/vimproc.vim

@justinmk After scratching my head about it for a while, I don't see any useful way to interface vimproc with syntastic. Assuming I decrypted it properly, vimproc's main use of backgrounding is about running several things in parallel and polling for results. For syntastic we'd need a mechanism to run a single process in background, and get an event when the process is done. These are pretty different use cases.

On a related topic: I played a little with AsyncCommand, and it isn't very robust. It appears to leave background processes behind, which can happily eat 99% of CPU cycles. And I wasn't even trying to crash it. :)

@lcd047 I'm surprised that vimproc doesn't fit this use case, but I haven't looked closely at it. AFAIK, there is no way to get passively notified of an external event in vim (without --remote which requires gvim). But using vim's &updatetime and CursorHold/CursorHoldI have been used by other plugins to reasonable effect. Why wouldn't that work for syntastic?

FWIW, I just bumped into another option: xolox's vim-misc combined with vim-shell. It is used by easytags to provided asynchronous behavior. See xolox#misc#os#exec(options) for usage of the vim-shell asynchronous capability.

Well, vim-dispatch mentioned above uses tmux to send Vim a SIGWINCH when a background command finishes. Also, I believe --remote requires +clientserver, not a GUI.

As for CursorHold, I know about people using it in creative ways to achieve async effects, but I never bothered to find out exactly how that works. If it's possible to emulate a poll loop that way, I'm not familiar with it. The burden of proof is on you, not me. :)

Anyway, personally I'm not going to waste any more brain cycles on this. Async make should be implemented in Vim, not emulated by some horrible contortions in third-party plugins. It's 2013, job control has been understood really, really well by now. Patching Vim to do that would be relatively straightforward --- except qf_init_ext() is ~700 lines long, and the chances of such a patch becoming official any time soon seem pretty slim. And, last but not least, I have exactly zero use for all this, I'm pretty happy with the sync checks. shrug

@lcd047 fair enough, I mostly agree.

For posterity: --remote does appear to require a GUI: https://code.google.com/p/macvim/issues/detail?id=431

@justinmk: Apparently it requires either OLE (that is, Windows), or an X-like clipboard (UNIX, Mac OS, or VMS). This means Vim needs to run in a (specific) GUI, but Vim itself doesn't need to be compiled with GUI support. I'm using this:

VIM - Vi IMproved 7.4b BETA (2013 Jul 28, compiled Jul 29 2013 09:10:27)
Compiled by root@euler
Huge version without GUI.  Features included (+) or not (-):

It can do remote commands as long as both the server and the client run in xterms, but not in console.

@lcd047 Very interesting! Good to know. It does work on Windows using any combination of {vim.exe, gvim.exe}.

Well, since it appears this github issue has become an accidental comprehensive survey of the state of "vim async support", perhaps it will save others the trouble in the future.

+1 Need async for javascript :)

@lcd047 : Any chance you could file a bug to AsyncCommand for how you produced abandoned background processes ("which can happily eat 99% of CPU cycles.")? AsyncCommand doesn't do any process killing, so if you do something like AsyncShell yes, then it will run forever (but isn't that what you'd expect)?

---

On topic:

Assuming I decrypted it properly, vimproc's main use of backgrounding is about running several things in parallel and polling for results. For syntastic we'd need a mechanism to run a single process in background, and get an event when the process is done. These are pretty different use cases.

Without using something like AsyncCommand (where you use --remote to send a command to vim), I don't think you can send an event, without polling for completion. So essentially, those are the same use cases except syntastic only fires one job. (Unless you mean vimproc blocks until all parallel jobs complete?)

I can't find the SIGWINCH call @lcd047 mentioned, but dispatch#callback is also using --remote. Maybe I'm missing something.

@tpope Wouldn't using syntastic and vim-dispatch's headless strategy make sense? I guess that would require adding a way to force a specific strategy to vim-dispatch? That seems like something that would be generally useful for other plugins to depend on too. (You could use it for an auto-exec repl, doc or tags lookup, and of course syntax checking.) Anywhere the user wants parsed output and not raw output.

@pydave I would keep an eye on tarruda's Vim fork to see where it goes. It's showing more promise than the floobits "timers" fork from October.

Any chance you could file a bug to AsyncCommand for how you produced abandoned background processes ("which can happily eat 99% of CPU cycles.")?

Vanishingly small chances. As I said above, this proved to be a huge time sink, and I'm not going to waste any more brain cycles on it. If you care about it, I suppose you can retrieve the relevant setup in #370 and do your own testing. If you revive that patch (or if you come up with some other working solution), people test it extensively, and it proves to be stable, then I'll gladly include it. Until then, I'd rather spend my time doing something else.

I can't find the SIGWINCH call @lcd047 mentioned

The term to look for is VimResized. SIGWINCH is the signal that makes it possible.

Hey guys, any further word on a solid solution for async checking? If not, what is the best (/ least terrible) 3rd party solution that you've come across? I've recently been converted from Xcode to Vim land (a very liberating move), yet the one thing I miss a lot is Xcode's excellent multi-file async checking. Syntastic is already incredible, but I just wanted to add another "with async it would be totally awesome!". I hope the passion for this feature hasn't died yet :-) Or even better, if this has already been addressed with a recent feature that I've somehow managed to miss, let me know!

If you're happy with the set of languages it supports, YouCompleteMe is pretty good at what it does. And if you're not married to Vim, InteliJ IDEA, PyCharm, and friends are rather useful, too.

@lcd047 Sorry for reviving this but I was wondering if you are following neovim development and if you can say to what degree the async stuff they have included will be of use for syntastic at some point ?

Sorry, I haven't followed neovim development recently. I'll probably look at it again when / if they actually produce some kind of release.

A look for the support if syntastic background checking led me to this ticket. Support for this would make syntastic even awesome'er!

Awhile back I found a fork of syntastic that tried to use idbrii/AsyncCommand to run syntax checking in the background. The initial and only commit stgpetrovic/syntastic-async@b3e3a9ba22b54aac6483e3b5ea545363d10fb304 (from a year ago) says:

First version that is workable with.

Will improve everything this is just the initial hack.

@s0undt3ch: if you're interested, you could try figuring out the problems with it. (Or whether it could work with vim-dispatch.)

(I haven't had time to look into VimResized that lcd047 mentioned above or actually mess with syntastic-async myself.)

I suppose I should update you guys on this. The short version is, it should be possible to make syntastic do async checks using vimproc alone, but there would be a lot of work involved, most checkers would have to be refactored to take advantage of it, and the result is likely to be pretty fragile. Also speed would likely suck for some checkers, despite them being run asynchronously. A longer version follows, if you care about that kind of details.

The discovery process went something like this: vimlint pointed me to vim-watchdogs, which does async checks. Its approach is very different from syntastic, so the trick there can't be applied directly to syntastic; however, it pointed me to vim-quickrun, which is essentially a higher level frontend to vimproc. This again isn't directly useful for syntastic, but digging through the sources reveals how to do polling for an external event from Vim, using CursorHold and feeding it invisible keystrokes with feedkeys(). So it would be possible (at least in principle) to run the checkers with vimproc, poll for results, and trigger syntastic notifications accordingly. This seems better than all other options listed above, for many technical reasons.

Now there are two main problems with this approach. First, we'd need an entire framework for dealing with tasks, processes, and queues. This can be done, but it's a lot of work, and the documentation for vimproc is less than useful.

Second, one can only run processes in background, not Vim functions. Since the results of most checkers are further munged by Vim before being put in the final loclist, dealing with this would involve a major rewrite of the entire checking mechanism. Also, having a single *_GetLocList() function per checker would no longer be enough, it would have to be split into a function that would submit the checker command line to vimproc, and another function that would post-process the results. Furthermore, this post-processing stage would be run in foreground (since it would be a Vim function, not a process), in a CursorHold handler. The same CursorHold handler would also be responsible for task bookkeeping. That is likely to suck, badly. A possible solution to speed things up would be to do the post-processing in something like Python instead of VimL, but that would mean re-writing all checkers to use Python, which would suck even worse, albeit in a different way.

Last but not least: using feedkeys() is almost guaranteed to be fragile. If you're a fast typist and you already run into Vim occasionally misreading some key combinations while you type, feedkeys() would complicate this problem for you exponentially. shrug

I mess up the screen pretty often trying to do stuff while syntastic is working. If there's a way to prevent that from happening maybe it's not such a big deal if the checkers are not async... just a thought.

@spacepluk As I explained in #822, this is a bug in your environment, and syntastic has absolutely no control over it.

@lcd047 thanks, that's very useful info.

I write an async version using clientserver feature. if anyone has interest, you can checkout my fork.

@q0LoCo I tried your async branch and it seems to work with flake8 for Python but not with ESLint for JavaScript. I'd open an issue there but you don't have the issue tracker enabled on your repository.