enrich
enrich copied to clipboard
snowplow
Snowplow Enrichment jobs and library
Migrated from https://github.com/snowplow/snowplow/issues/1912
At the moment the creation of the source and the sinks [are independent](https://github.com/snowplow/enrich/blob/3.2.5/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Run.scala#L51-L54) and as a consequence they can't share resources (e.g. clients). For instance for Kinesis a client is...
Once #649 has been solved, we will correctly recover from exceptions when a worker loses a lease. But.... immediately after losing a release, there is a high chance that we...
The function orchestrating enrich [returns a F[Unit]](https://github.com/snowplow/enrich/blob/3.1.5/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Enrich.scala#L263-L266) instead of a `Stream[F, Unit]`. This results in using `.compile.drain` in 2 different places ([here](https://github.com/snowplow/enrich/blob/master/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Run.scala#L168) and [here](https://github.com/snowplow/enrich/blob/master/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Enrich.scala#L271-L272)) instead of once. Streaming transformer [does...
At the moments there are 2 different levels of backoff retry : - One in case `PutRecords` request fails, orchestrated by `cats-retry` - One in case `PutRecords` succeeds but contains...
Follow up from https://github.com/snowplow/enrich/issues/621.
When troubleshooting and/or benchmarking it's very useful to be able to know how long it takes to enrich and to sink a chunk
Exception seen on `Sentry` for enrich 3.1.3: ``` java.lang.NoClassDefFoundError: Could not initialize class org.joda.time.format.ISODateTimeFormat$Constants at org.joda.time.format.ISODateTimeFormat.dateTime(ISODateTimeFormat.java:775) at org.joda.time.base.AbstractInstant.toString(AbstractInstant.java:424) at org.joda.time.base.AbstractDateTime.toString(AbstractDateTime.java:314) at com.snowplowanalytics.snowplow.badrows.package$.$anonfun$dateTimeEncoder$1(package.scala:25) at io.circe.Encoder$$anon$1.apply(Encoder.scala:55) at io.circe.Encoder$$anon$21.apply(Encoder.scala:314) at io.circe.Encoder$$anon$21.apply(Encoder.scala:312) at com.snowplowanalytics.snowplow.badrows.Payload$RawEvent$anon$lazy$macro$29$1$$anon$5.encodeObject(Payload.scala:111)...
Although all enrich assets are now fully streaming, weather enrichment still uses [historical](https://openweathermap.org/history) API and as a result it can't be used any more. [Current](https://openweathermap.org/current) API needs to be used...