replicate-javascript icon indicating copy to clipboard operation
replicate-javascript copied to clipboard

Add "strict" interface to `replicate.run()`

Open aron opened this issue 1 year ago • 0 comments

This PR implements a new strict option that can be passed to run() to coerce the output into the same data types as defined by the model schema.

const stream = replicate.run("my-org/my-llm", {strict: true, input: { ... }});
for await (const chunk in stream) {
  ...
}

const files = replicate.run("my-org/my-gen-img", {strict: true, input: { ... }});
for await (const image in files) {
  const img = Object.createURLObject(await file.blob());
}

const file = replicate.run("my-org/my-model", {strict: true, input: { ... }});
const data = await file.arrayBuffer();

The main two benefits here are that:

  1. Iterators get converted into actual async iterators instead of just arrays which means the stream() method can be deprecated in favor of a single interface. This opens up opportunities for streaming other types of output too.
  2. Path & File outputs are backed by a Blob like representation that can be used to download the data opening up opportunities to stream files directly in the client. Note, that we don't use Blob, File or ArrayBuffer directly because it's likely the client will want the opportunity to decide whether to actually download the file or not. Similar to how body on Response is not consumed until an async read method is called. Feedback on this appreciated.

[!NOTE] The intention here is that once this code is merged and validated we migrate the SDKs to use this as the default interface to run() (a breaking change). The strict flag is a way of soft launching the feature. The name strict was chosen as in JavaScript land "use strict" and TypeScript --strict mode are existing functionality for tightening up interfaces.

The need to download the schema for the version is currently unfortunate, I'm also looking at extending the Prediction interface to include input_schema and output_schema to reduce this additional request.

Implementation

The bulk of the code exists in the coerceOutput() function, I think it implements the bulk of the Cog Output interface, but I may have missed some edge cases.

TODO

  • [ ] Currently there is no support for enum types.
  • [ ] Currently there is no support for date strings.

aron avatar Jul 08 '24 13:07 aron