Can I add a Livy server docker image to this stack?

[ericdill]

Hey @jcrist ,

Would you review and ultimately accept a PR that added Livy server to this hadoop stack?

[jcrist]

Yes, that'd be very welcome. I just redid the docker images so this should be a more tenable task now.

We now have a single Dockerfile with an ARG for which version of CDH to build. For the versions we care about this produces 2 images:

• jcrist/hadoop-testing-cdh5
• jcrist/hadoop-testing-cdh6

Both of these contain support for optional kerberos security based on a runtime envvar.

To extend with additional functionality, you'd create a new Dockerfile based on one of these images, install the required dependencies, copy in the required config, and copy in a supervisord config file for wherever the livy server should run (either edge or master I suspect?). This should just work with the htcluster command then:

htcluster startup --image your-image --config kerberos

I'd be happy to help get this working if you're still interested.

[danielfrg]

Eric LOOOOOVES Livy.

[ericdill]

ISTM i'll need to do the following:

• [ ] add a new host, livy.example.com?
• [ ] add livy and HTTP users and keytabs for livy.example.com
• [ ] download and unpack livy zip
• [ ] create livy configuration files that specify, among other things, the user / keytabs for livy/HTTP
• [ ] add a livy service to the docker-compose.yml
• [ ] update hadoop XMLs to allow livy to impersonate users
• [ ] add supervisor.d stuff for livy