weave icon indicating copy to clipboard operation
weave copied to clipboard

feat(weave): Adds Dataset.map and progress bars

Open tcapelle opened this issue 7 months ago • 4 comments

This PR replaces #3751

  • Modifies async_foreach to support rich progressbar
  • Enables weave.Evaluation with progress bar, also verbose=False to silence printing per row
  • Adds Dataset.map that uses async_foreach to process each example
  • Same inspect convention as with evaluations.
ds = weave.Dataset(rows=[
    {"id": i, "val": i} for i in range(1000)
])

@weave.op
def double_value(val):
    time.sleep(.001)  # Simulate work
    return {"new": val * 2}

new_ds1 = asyncio.run(ds.map(double_value))

tcapelle avatar May 05 '25 08:05 tcapelle

@andrewtruong as I added a method to dataset, is it normal that the hashes changes? (and thus the failing tests)

tcapelle avatar May 12 '25 15:05 tcapelle

Yes. Since you added a new op, the digest is expected to change

andrewtruong avatar May 12 '25 15:05 andrewtruong

Yes. Since you added a new op, the digest is expected to change

How do I fix the tests then?

tcapelle avatar May 20 '25 09:05 tcapelle