streamparse
streamparse copied to clipboard
pystorm
Support selection of Python implementation on an individual component.
Given the use of ShellBolt, we could consider having a given component in the streamparse-based topology use a Python implementation other than the one that streamparse is using. This could be useful e.g. to have one component running in PyPy while others are running Python 2 and Python 3 simultaneously.
I like this idea, but it would require separate virtualenvs per component in the topology. We're going to end up having to do that anyway to avoid race conditions when we switch to using the topology hook to address #99, but it's worth noting.
Good point. The configuration for this could point to paths for the Python executables within the virtualenvs, as to decouple virtualenv management from the config itself.
path/to/virtualenv/bin/python
I was thinking more along the lines of having the config specify the Python version needed and then have the topology hook install the appropriate version.
It seems a little simpler to the user to have them just specify the version and not make them care about where things actually end up on the servers (especially since they won't be SSHing into them anymore).
That makes sense. It'd be great to at least discuss in docs how to configure this. It might be as simple as making sure the $PATH is correct in finding Python executables by name. I generally prefer being able to configure this things, but as long as it's clear how to do it, I'm +1.
Since we now support component-specific config dictionaries via the Python Topology DSL, I think this could probably just be a special setting, maybe called streamparse.python.version. That way people could specify that in their storm.yaml (once we switch to using that instead of config.json) to have it work for all components, and it could be overridden for specific components.
I prefer streamparse.python.path or streamparse.python.executable to allow referencing any Python implementation. If we do want streamparse.python.version, then it would help to expose the mapping of versions to command-line invocation to allow full control.
The idea is that each component would have its own separate conda environment (or maybe PEX as in #212), so there wouldn't be one path to refer to.
Sounds interesting. However we do it, I'm advocating that we support "Bring Your Own Python." If we do streamparse.python.version, a user will look for a way to extend the mapping of version identifiers to Python implementations (in whatever way makes sense to the configuration of streamparse).