Match vega backend capabilities with matplotlib

[janpipek]

And while doing so, extract as much shared code as possible.