scio icon indicating copy to clipboard operation
scio copied to clipboard

Published 20 hours ago • verified publisher icon spotify

Add SparkeyIO

Open kellen opened this issue 3 years ago • 4 comments

Adds SparkeyIO, makes sparkey endpoints testable in JobTest. See #3173

Open questions:

  • Should SparkeyIO be used on the write path and if so how?
  • Is there a better way to pass inputs from JobTests than SparkeyIO.testUri(...)?

kellen avatar Jul 22 '20 21:07 kellen

@psobot thoughts? I like the idea. Naming of SparkeyIO etc. can be tweaked a bit but overall this could be useful?

nevillelyh avatar Jul 23 '20 18:07 nevillelyh

So what should we do next? It is a bit hacky but also solves a practical need. Also it's in scio-extra so I think we can live with it as long as we avoid confusing naming. @regadas?

nevillelyh avatar Aug 03 '20 14:08 nevillelyh

I would like to give this some thought. I agree, it's practical but in scio-extra this is probably the most used io/sideInput and for that reason I'm a little bit hesitant in adding things that we might discard in a near future.

regadas avatar Aug 04 '20 10:08 regadas

@regadas any ideas on how we might restructure this to fit better w/ the rest of the architecture?

kellen avatar Sep 14 '20 22:09 kellen

Closing in favor of a different approach

kellen avatar Dec 15 '23 16:12 kellen