Connecting EC2 instance to EMR Livy enabled spark cluster

[ggittu]

@devstein We are experimenting the connection from EC2 instance (where Jupiter is setup with sparkmagic) to EMR spark cluster(Livy enabled). I am following this blog post https://towardsdatascience.com/every-data-scientist-can-need-some-spark-magic-853da0b81006. But unfortunately its not explained well the connecting part from the Jupyter book to the EMR cluster.

Is there any documentation I can follow for finishing the setup ?

[clayms]

@ggittu Are you not able to Add an End Point and then Create a Session?

Try the following to connect to your EMR cluster

1. Add an Endpoint (replacing 10.20.40.50 with your EMR master-node-ip):

2. Create the Session

3. Check the Session