enrich icon indicating copy to clipboard operation
enrich copied to clipboard

Remove the possibility to emit enriched events invalid against atomic schema

Open benjben opened this issue 3 years ago • 0 comments

Remove the feature flag acceptInvalid in the configuration and consider it as always false.

acceptInvalid and invalidCount can be removed from these 2 functions :

  1. In EnrichmentManager :
def enrichEvent[F[_]: Monad: RegistryLookup: Clock](
    registry: EnrichmentRegistry[F],
    client: Client[F, Json],
    processor: Processor,
    etlTstamp: DateTime,
    raw: RawEvent,
    acceptInvalid: Boolean,
    invalidCount: F[Unit]
  ): EitherT[F, BadRow, EnrichedEvent] =
  1. In EtlPipeline :
 def processEvents[F[_]: Monad: RegistryLookup: Clock: HttpClient](
    adapterRegistry: AdapterRegistry,
    enrichmentRegistry: EnrichmentRegistry[F],
    client: Client[F, Json],
    processor: Processor,
    etlTstamp: DateTime,
    input: ValidatedNel[BadRow, Option[CollectorPayload]],
    acceptInvalid: Boolean,
    invalidCount: F[Unit]
  ): F[List[Validated[BadRow, EnrichedEvent]]] =

More details here.

benjben avatar Feb 09 '22 16:02 benjben