weave
weave copied to clipboard
feat(weave): Adds Dataset.map and progress bars
This PR replaces #3751
- Modifies
async_foreachto support rich progressbar - Enables
weave.Evaluationwith progress bar, alsoverbose=Falseto silence printing per row - Adds
Dataset.mapthat usesasync_foreachto process each example - Same
inspectconvention as with evaluations.
ds = weave.Dataset(rows=[
{"id": i, "val": i} for i in range(1000)
])
@weave.op
def double_value(val):
time.sleep(.001) # Simulate work
return {"new": val * 2}
new_ds1 = asyncio.run(ds.map(double_value))
Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=febc734b177472f5b13a1a2d33f5cd0cd39d6d19
@andrewtruong as I added a method to dataset, is it normal that the hashes changes? (and thus the failing tests)
Yes. Since you added a new op, the digest is expected to change
Yes. Since you added a new
op, the digest is expected to change
How do I fix the tests then?