scio icon indicating copy to clipboard operation
scio copied to clipboard

Scala API for Scio Transforms

Open anish749 opened this issue 4 years ago • 1 comments

We have some awesome transforms in Scio's transforms package. When using them in Scala pipelines, it can be less idiomatic. It would be nice to have first class support for this.

I was using RateLimiterDoFn and DoFnWithResource and I was hoping to have something like these on an SCollection[T]

def mapWithResource[R, U: Coder](resource: => R, resourceType: ResourceType)(fn: (R, T) => U): SCollection[U]

or

def rateLimitPerWorker(recordsPerSecond: Double): SCollection[T]

I can send a PR with some of the stuff if it fits in here.

anish749 avatar Jul 03 '20 08:07 anish749

@zwycl will take a stab at this.

regadas avatar Sep 29 '20 16:09 regadas

Since https://github.com/spotify/scio/pull/3389 is merged, should this task be closed?

l0kr avatar Oct 10 '22 16:10 l0kr