Ebc demo

[keith-mcclellan]

Conflicts:

cli_script.sh

[keith-mcclellan]

Not worried about the readme changes, more interested in getting stage_ebc_demo.sh into master.  Thanks!!

--  Keith McClellan Technical Lead, Federal Programs at Mesosphere [email protected] (202) 683-8558 direct

On September 20, 2016 at 1:19:19 PM, Michael Ellenburg ([email protected]) wrote:

@mellenburg requested changes on this pull request.

@keith-mcclellan I think this PR should be contained to a few README.md updates. @orsenthil just embellished this a bit, so I'll try merging your changes in light of his and we can find something that covers this material

In README.md:

-Run the Load Dependencies step to load the required libraries into Zeppelin. Next, run the Spark Streaming step, which reads the tweet stream from Zookeeper, and puts them into a temporary table that can be queried using SparkSQL. Next, run the Top Tweeters SQL query, which counts the number of tweets per user, using the table created in the previous step. The table updates continuously as new tweets come in, so re-running the query will produce a different result every time.

-NOTE: if /service/zeppelin is showing as Disconnected (and hence can’t load the notebook), make sure you're using HTTPS instead of HTTP, until this PR gets merged. Alternatively, you can use marathon-lb. To do this, add the following labels to the Zeppelin service and restart:

-HAPROXY_0_VHOST = [elb hostname]

-HAPROXY_GROUP = external

-You can get the ELB hostname from the CCM “Public Server” link. Once Zeppelin restarts, this should allow you to use that link to reach the Zeppelin GUI in “connected” mode.

-## Developing Tweeter I don't think we want to remove the instructions on how to actually build the tweeter app

In README.md:

-Install package CLIs: +Run bash stage_ebc_demo.sh [master_ip] [ELB_Hostname] The README should point to cli_script.sh which has all of the options

In README.md:

-## Post a lot of Tweets

-Post a lot of Shakespeare tweets from a file:

-```
-dcos marathon app add post-tweets.json

-```

-This will post more than 100k tweets one by one, so you'll see them coming in steadily when you refresh the page. Take a look at the Networking page on the UI to see the load balancing in action.

-## Streaming Analytics

-Next, we'll do real-time analytics on the stream of tweets coming in from Kafka. +* Update your Python Env to work with the latest CLI +* Install a local CLI at ./dcos This script core does not update local pyenv nor does it install the base CLI (it does install a few package CLIs)

In README.md:

-``` -$ dcos kafka connection -... -$ dcos cassandra connection -...

-```

-## Edit the Tweeter Service Config

-Edit the HAPROXY_0_VHOST label in tweeter.json to match your public ELB hostname. Be sure to remove the leading http:// and the trailing / For example:

-```json -{

• "labels": {
• "HAPROXY_0_VHOST": "brenden-7-publicsl-1dnroe89snjkq-221614774.us-west-2.elb.amazonaws.com" I believe this step is still necessary if you want to have hostname resolution for the IP instead of IP. The CLI script doesnt use it is not strictly necessary and it is nontrivial to automatically detect the hostname

In stage_ebc_demo.sh:

+# Alt Usage: bash stage_ebc_demo.sh [masterip] [publicELB] + +# Requirements: +# - DC/OS cluster with 1 public slave and 5 private slaves with +# or without superuser set +# - DCOS CLI installed on localhost +# - DCOS_EE set to true or false +# - DCOS_URL set to DCOS master URL +# +# If no user credentials are supplied, the following will be used: +# Enterprise: +# - AWS default bootstrapuser/deleteme +# - Override with DCOS_USER & DCOS_PW +# OSS: +# - The token hard-coded below for [email protected] +# password: unicornsarereal This information is no longer accurate. CI account changed. Best to not have two doc's sections describing the same thing

In stage_ebc_demo.sh:

+# - Override with DCOS_AUTH_TOKEN +set -o errexit + +if [ -z ${DCOS_EE+x} ]; then DCOS_EE=true; fi +if [ -z ${DCOS_URL+x} ]; then +#strip http(s) from Master IP url +mip_clean=${1#//} +#strip trailing slash from Master IP url +DCOS_URL=http://${mip_clean%/} +fi + +if [ -z ${DCOS_PUB_ELB+x} ]; then +# strip http(s) from ELB url +elb_clean=${2#//} +# strip trailing slash from ELB url +DCOS_PUB_ELB=${elb_clean%/} This variable is never used

In stage_ebc_demo.sh:

+# or without superuser set +# - DCOS CLI installed on localhost +# - DCOS_EE set to true or false +# - DCOS_URL set to DCOS master URL +# +# If no user credentials are supplied, the following will be used: +# Enterprise: +# - AWS default bootstrapuser/deleteme +# - Override with DCOS_USER & DCOS_PW +# OSS: +# - The token hard-coded below for [email protected] +# password: unicornsarereal +# - Override with DCOS_AUTH_TOKEN +set -o errexit + +if [ -z ${DCOS_EE+x} ]; then DCOS_EE=true; fi This doesn't need to be set

In stage_ebc_demo.sh:

+# If no user credentials are supplied, the following will be used: +# Enterprise: +# - AWS default bootstrapuser/deleteme +# - Override with DCOS_USER & DCOS_PW +# OSS: +# - The token hard-coded below for [email protected] +# password: unicornsarereal +# - Override with DCOS_AUTH_TOKEN +set -o errexit + +if [ -z ${DCOS_EE+x} ]; then DCOS_EE=true; fi +if [ -z ${DCOS_URL+x} ]; then +#strip http(s) from Master IP url +mip_clean=${1#*//} +#strip trailing slash from Master IP url +DCOS_URL=http://${mip_clean%/} I do not understand the point of removing http(s):// and then immediately tacking it back on. The CLI can handle either case

In stage_ebc_demo.sh:

+#strip trailing slash from Master IP url +DCOS_URL=http://${mip_clean%/} +fi + +if [ -z ${DCOS_PUB_ELB+x} ]; then +# strip http(s) from ELB url +elb_clean=${2#*//} +# strip trailing slash from ELB url +DCOS_PUB_ELB=${elb_clean%/} +fi + +echo $DCOS_URL +echo $DCOS_PUB_ELB + +#Run CI Script in infrastructure mode +bash cli_script.sh --url $DCOS_URL --infra If the whole purpose of the script is to take a set DCOS_URL and change it from https to http, then I don't think this script should be in the repo. Perhaps instead just have an example line in the README that says.

Stage EBC Demo: bash cli_script.sh --infra --url http://my.dcos.url — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[keith-mcclellan]

@mellenburg can you please review my change to the cli_script.sh? Should now properly determine that cassandra is ACTUALLY up before launching the tweeter front end, instead of just the cassandra scheduler. this change is required due to changes in the v2 framework