scio
scio copied to clipboard
Scala API for Scio Transforms
We have some awesome transforms in Scio's transforms package. When using them in Scala pipelines, it can be less idiomatic. It would be nice to have first class support for this.
I was using RateLimiterDoFn
and DoFnWithResource
and I was hoping to have something like these on an SCollection[T]
def mapWithResource[R, U: Coder](resource: => R, resourceType: ResourceType)(fn: (R, T) => U): SCollection[U]
or
def rateLimitPerWorker(recordsPerSecond: Double): SCollection[T]
I can send a PR with some of the stuff if it fits in here.
@zwycl will take a stab at this.
Since https://github.com/spotify/scio/pull/3389 is merged, should this task be closed?