Feature request: prediction streaming

[daneden]

Hi there! I’ve recently started playing with Replicate in SwiftUI and this library has been super helpful.

In my previous custom implementation of the Replicate HTTP API, I was doing some manual polling to get a stream of updates in predictions:

// Illustrative pseudocode
Task {
  while prediction.completedAt == nil {
    do {
      try await Task.sleep(nanoseconds: 1_000_000_000)
      prediction = try await fetchPrediction(prediction.id)
    } catch {
      print(error)
    }
  }
}

// 1 second
print(prediction.output) // ["Hello", "!", " "]

// 2 seconds
print(prediction.output) // ["Hello", "!", " ", "As", " ", "a", " ", "helpful", ...]

It'd be great if this library included a polling/streaming method alongside the wait method, especially for chat-like models.

[mattt]

Hi @daneden. That's a great idea! We recently added support for streaming prediction output to Replicate. #41 added an overload to Client.createPrediction that accepts a stream parameter. When set to true, the created prediction will include a "stream" key in its urls property.

My next step is to create a Swift implementation of EventSource that conforms to AsyncSequence and you could iterate over directly.

[tianye2856]

Please support Streaming and EventSource as the Replicate JavaScript code you provided here. https://github.com/replicate/replicate-javascript#streaming

Now even that you have a stream parameter, it doesn't make any sense. Because we still need to make the stream logic code by ourself.