flambo
flambo copied to clipboard
Various SQL, api and context functionality adds
This pull request includes:
- Scala -> Clojure recursive data type conversion
- SQL array type support for
StructType
creation - Various other small utility functions
- Context creation/caching.
The last one I might need your help. I broke it out into a separate namespace since it seemed to need its own or I thought you might specifically have a place for it. I also thought you'd end up wanting to refactor it quiet a bit.
The idea here is that you only get one Spark context per JVM, so might as well store it in an atom
and keep it around without having to create it over and over. This esp. true since most of the initial configuration (expect the app name, which I'm open for suggestions) can be derived from the JVM environment (i.e. properties etc).