snowplow-rdb-loader
snowplow-rdb-loader copied to clipboard
snowplow
Stores Snowplow enriched events in Redshift, Snowflake and Databricks
If user saves the data at the root of the bucket, like: ``` s3://snowplow-bucket/run=2022-01-28-16-30-00/ ``` Instead of: ``` s3://snowplow-bucket/shredded/run=2022-01-28-16-30-00/ ``` Our algorithm fails to figure out the bucket is processed...
We have seen this exception from RDB loader 1.1.1, but we don't yet know the cause: > SQL `NULL` read at column 1 (JDBC type Char) but mapping is to...
Currently, region config decoder is using `getRegion` method from `DefaultAwsRegionProviderChain`. Since this method is accessing to environment, it isn't pure in functional programming sense. Decoders should be completely pure therefore...
This is a follow-up for #608. In addition to just checking if loading is progressed via stages, we also should be able to leverage [`STV_LOAD_STATE`](https://docs.aws.amazon.com/redshift/latest/dg/r_STV_LOAD_STATE.html) docs claim it's available to...
Currently we use our own [`Logging` algebra](https://github.com/snowplow/snowplow-rdb-loader/blob/a2bc768803115b15649f9518fc59d30d1b0f50a4/modules/loader/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/dsl/Logging.scala#L22), which allows us to test logging and abstract it away, however it's very unflexible. 1. It doesn't allow to disable logging for some...
https://spark.apache.org/docs/latest/monitoring.html#metrics sister ticket to snowplow/snowplow#3514
In a case where a batch has two schemas that would get transformed into the same table, (e.g. `some-schema` and `some_schema`) we produce not-so-helpful error message: https://github.com/snowplow/snowplow-rdb-loader/blob/a2bc768803115b15649f9518fc59d30d1b0f50a4/modules/loader/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/db/Migration.scala#L251 At very least...
We have tens of occurences of this error for different clients: ``` com.snowplowanalytics.snowplow.rdbloader.LoaderError$StorageTargetError: Database error: [Amazon](500310) Invalid operation: 1023 Details: Serializable isolation violation on table - 100167, transactions forming the...
Whether it is single or multiple loader running together on the same host, only requirement on ssh tunnel's local port is not being allocated on the host the app is...
