telemetry-analysis-service
telemetry-analysis-service copied to clipboard
mozilla
Telemetry Analysis Service
See https://bugzilla.mozilla.org/show_bug.cgi?id=1324464
I launched two clusters yesterday, and neither was able to successfully run a notebook. When opening a notebook, both stuck in the "Kernel starting, please wait..." state and could not...
Quoting from https://bugzilla.mozilla.org/show_bug.cgi?id=1312747: > Currently airflow and atmo are using two different EMR steps [1] [2] for almost the same logic. We should refactor those into a single script and...
From https://bugzilla.mozilla.org/show_bug.cgi?id=1373631: Py4JJavaErrorTraceback (most recent call last) in () ----> 1 serialized_beta_full[1].count() /usr/lib/spark/python/pyspark/rdd.py in count(self) 1006 3 1007 """ -> 1008 return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum() 1009...
From https://bugzilla.mozilla.org/show_bug.cgi?id=1373633 Sequence of events: In Spark: serialized_beta_full[1].saveAsTextFile("s3://net-mozaws-prod-us-west-2-pipeline-analysis/ekr/serialized-beta-full.out") In hadoop: hadoop fs -getmerge s3://net-mozaws-prod-us-west-2-pipeline-analysis/ekr/serialized-beta-full.out serialized-beta-full.out This claims to copy a lot of files, but the result is 0-length.
We want datadog to send a notifier when the number of Sentry exceptions in a day spikes.
AWS Cloudwatch recently gained the ability (thanks @robotblake!) to [stream EMR events](http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-cloudwatch-events.html). That means that cluster state changes can be forwarded to the available EMR targets, including custom Lambda functions...
For development and staging we should use separate S3 buckets and research of there are other places of overlap.
We are running into some problems with spot instances: 1. Our pricing (and limited machine type/region) is really killing the market. Pricing is up 2x at some points, and we...
