snowplow-rdb-loader
snowplow-rdb-loader copied to clipboard
snowplow
Stores Snowplow enriched events in Redshift, Snowflake and Databricks
We were shown [an example on Discourse](https://discourse.snowplow.io/t/rdb-loader-failing-with-varying-schemas/7562) where the Databricks Loader seemed to get stuck running `SELECT 1` when it first started up. In the loader [we use JDBC query...
The `Statement` sealed trait has [a few statements for transient tables](https://github.com/snowplow/snowplow-rdb-loader/blob/4.3.0/modules/loader/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/db/Statement.scala#L87-L89). As far as I can tell they are never used, so we should get rid of these objects.
RDB loader sends alerts via a webhook to a listener, [configured here](https://github.com/snowplow/snowplow-rdb-loader/blob/4.3.0/config/redshift.config.reference.hocon#L65-L72). It is expected that every payload is an actionable alert, to be addressed by the pipeline operator. There...
At the moment, micro batches in the window are created according to [their count](https://github.com/snowplow/snowplow-rdb-loader/blob/master/modules/transformer-kinesis/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/transformer/kinesis/sinks/generic/Partitioned.scala#L126). However, this approach would lead to memory problems with oversized events. Instead, these micro batches should...
In Snowflake Loader, Snowflake host address is constructed with Snowflake region and Snowflake account name at the moment. However, [new Snowflake account name format](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#format-1-preferred-account-name-in-your-organization) allows to create URLs without Snowflake...
This is suggested by open source customer in [here](https://discourse.snowplowanalytics.com/t/retry-for-rdb-stream-transformer/7229). In a nutshell, S3 throws some ephemeral 50x exceptions in some cases. Instead of reprocessing the window, writing it to s3...
The transformer config file has a `skip` option which [is documented as](https://github.com/snowplow/snowplow-rdb-loader/blob/4.2.1/config/transformer.batch.config.reference.hocon#L66): ``` # Schemas that won't be loaded # Optional, default value [] "skip": [ "iglu:com.acme/skip-event/jsonschema/1-*-*" ] ``` If...
Currently the loader and transformer apps only accept base64-encoded configuration passed directly on the command line: ``` --config= --iglu-config= ``` However, all other Snowplow applications allow passing configuration from files:...
We often see that a warehouse can become mis-configured unexpectedly. For example, a warehouse admin might remove a permission from the loader role, which then prevents the loader from loading...
In case `atomic.events` table does not exist when loader starts, it should try to create it, using [this definition](https://github.com/snowplow/snowplow/blob/master/4-storage/redshift-storage/sql/atomic-def.sql). If table doesn't exist and loader can't create it, loader should...
