enrich
enrich copied to clipboard
snowplow
Snowplow Enrichment jobs and library
Full write-up: https://docs.google.com/spreadsheets/d/1UaXrH92IvRWyXNU8wUQ-oxvEI9kJxoxbIcbRjna7RAI/edit?usp=sharing * [ ] More unit tests * [ ] More enrichments (at least cover all Context+Enrichment all above spreadsheet) * [ ] Better schemaing for new contexts...
[Configured timeout](https://github.com/snowplow/enrich/blob/master/config/enrichments/api_request_enrichment_config.json#L36) for API enrichment, that ends up [here](https://github.com/snowplow/enrich/blob/3.7.0/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/enrichments/registry/apirequest/HttpApi.scala#L33) is never used. The HTTP client used by the enrichment is injected implicitly ([here](https://github.com/snowplow/enrich/blob/3.7.0/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/enrichments/registry/apirequest/ApiRequestEnrichment.scala#L237), [here](https://github.com/snowplow/enrich/blob/3.7.0/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/utils/HttpClient.scala#L47) and [here](https://github.com/snowplow/enrich/blob/3.7.0/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Environment.scala#L195)). This client [uses the...
In https://github.com/snowplow/enrich/commit/ec6fe746b5b00a8a2c6c31920c27c3725269aab8 we introduced the possibility to use environment variables for: - application hocon - Iglu resolver configuration - enrichments configuration files It'd be great to add an automated test...
In order to refactor atomic events we need to extract all non-generic information from a fat table into dedicated contexts and preserve only common properties. As a first step, we...
The config of the currency conversion enrichment exposes [rateAt](https://github.com/snowplow/enrich/blob/3.7.0/config/enrichments/currency_conversion_config.json#L13) but it is [never used](https://github.com/snowplow/enrich/blob/3.7.0/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/enrichments/registry/CurrencyConversionEnrichment.scala#L50-L82). Under the hood, the enrichment uses [scala-forex](https://github.com/snowplow/scala-forex) (which is based on [Open Exchange](https://openexchangerates.org/)). The API allows...
In our [documentation](https://docs.snowplow.io/docs/understanding-tracking-design/understanding-schemas-and-validation/) we express a firm position that for self-describing data, the top-level field containing the data must be an object. In other words, we can't define an event...
If [UserAgentAnalyzer.parse()](https://github.com/snowplow/enrich/blob/3.6.0/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/enrichments/registry/YauaaEnrichment.scala#L113) throws an exception, it is not caught directly and a generic error bad row is emitted ([here](https://github.com/snowplow/enrich/blob/3.6.0/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Enrich.scala#L145)). Example exception: ``` java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length...
The idea comes [from this discussion](https://github.com/snowplow/enrich/pull/678#discussion_r1010378090). Currently we have a lot of Iglu Central schemas hard-coded into the Enrich repo, e.g. [here](https://github.com/snowplow/enrich/tree/3.5.0/modules/common-fs2/src/test/resources/iglu-client-embedded/schemas). We even have duplicates where different tests depend...
PII = Personally Identifiable Information The basic idea: - Any JSON Schema (ue or context) can be annotated with `"pii": true` on a per-property basis - If this PII Scrubber...