Make debugging and tracing a constant theme in the Onboarding stories

[dsabeti]

A million years ago when @oozie went through CF Onboarding, he gave me the following feedback:

Inspired by our discussion in the retro, regarding bosh proficiency at the end of the course, i.e. giving the people a fishing rod over a fish, we had this idea: Make debugging/tracing a theme for working with every component like in https://www.pivotaltracker.com/story/show/151143904

We have stories where the users ssh onto BOSH VMs or App containers. I'm wondering if there are other ways to add more debugging and tracing skills into the stories. For example,

• Customize your bosh director with bbl
• Give users an ops-file that doesn't quite work and follow the error messages to learn how to fix it

What are some other ways we can incorporate more debugging and tracing?

[dsabeti]

We could try to write stories that cover things like:

1. bosh ssh XYZ -c ... to perform a command on each VM for a given instance group XYZ
2. Fetch logs for jobs, either by ssh'ing or running bosh logs. For example, use CF_TRACE during a cf push, and search the relevant jobs for the request guids.
3. Using cf events to understand what's changing about your app.
4. CF CLI plugins that provide information about the app.
5. What does each line in the output of cf push mean?
6. We've heard that CAPI has a fire drill for debugging the app. Maybe we can use that?

[ljfranklin]

A story mentioning VCAP_REQUEST_ID might be useful

[oozie]

I'll throw in my go-to tracing tricks:

• CF CLI export CF_TRACE=true
• curl -v and openssl for troubleshooting certificate info

curl -v https://www.google.com
openssl s_client -connect www.google.com:443

• GODEBUG for go-based projects env GODEBUG=gctrace=1,schedtrace=1000 godoc -http=:8080
• GRPC clients

export GRPC_TRACE=all
export GRPC_VERBOSITY=DEBUG 

• Ruby tracer for Ruby based software