Support stream: true for the completions endpoint

[petergoldstein]

The completions endpoint supports a streaming configuration, which may be valuable in situations where low latency is a priority and a stream of partial values to the caller is useful (i.e. for a chat client). This configuration uses server-sent events (SSEs) to incrementally provide updates to the caller.

I'm not entirely sure how to use the underlying HTTParty stack with server-sent events, assuming that's even possible. Needs further research and thought.

[bf4]

It seems you might be able to by setting the HTTParty connection adapter to support the HTTP gem and then perhaps use https://github.com/launchdarkly/ruby-eventsource which I see you've worked on?

[petergoldstein]

@bf4 Interesting suggestion. I don't have time to work on it right now, but I'd welcome a PR if you're so inclined.

[bf4]

Took a stab at running through the connection set up.. not sure if the difference in assumption with how Net::HTTP builds a connection with the uri, while HTTP waits until the actual request is easy to resolve. Faraday wrapping HTTP might be a better match since the adapter needs to mimic Net::HTTP, but otherwise, got pretty close, I think.

investigative spike

    def connection
      # https://github.com/jnunemaker/httparty/blob/30f2ec15917ef75934e7e7b139d9c92c22141093/lib/httparty/connection_adapter.rb
      # https://github.com/httprb/http/blob/462d711040735be3661068e9e52eb14a4269f1f4/spec/lib/http_spec.rb#L14
      # HTTP::Client.new
      http = ::HTTP
      # host = clean_host(uri.host)
      # port = uri.port || (uri.scheme == 'https' ? 443 : 80)
      if options.key?(:http_proxyaddr)
        http = http.via(
          options[:http_proxyaddr],
          options[:http_proxyport],
          options[:http_proxyuser],
          options[:http_proxypass]
        )
      #   http = Net::HTTP.new(
      #     host,
      #     port,
      #     options[:http_proxyaddr],
      #     options[:http_proxyport],
      #     options[:http_proxyuser],
      #     options[:http_proxypass]
      #   )
      # else
      #   http = Net::HTTP.new(host, port)
      end

      # http.use_ssl = ssl_implied?(uri)

      attach_ssl_certificates(http, options)

      timeout_options = {}
      if add_timeout?(options[:timeout])
        # http.open_timeout = options[:timeout]
        # http.read_timeout = options[:timeout]
        timeout_options[:connect] = options[:timeout]
        timeout_options[:read] = options[:timeout]

        from_ruby_version('2.6.0', option: :write_timeout, warn: false) do
          # http.write_timeout = options[:timeout]
          timeout_options[:write] = options[:timeout]
        end
      end

      if add_timeout?(options[:read_timeout])
        # http.read_timeout = options[:read_timeout]
        timeout_options[:read] = options[:read_timeout]
      end

      if add_timeout?(options[:open_timeout])
        # http.open_timeout = options[:open_timeout]
        timeout_options[:connect] = options[:open_timeout]
      end

      if add_timeout?(options[:write_timeout])
        from_ruby_version('2.6.0', option: :write_timeout) do
          # http.write_timeout = options[:write_timeout]
          timeout_options[:write] = options[:write_timeout]
        end
      end
      if timeout_options.any?
        http = http.timeout(**timeout_options)
      end

      # if add_max_retries?(options[:max_retries])
      #   from_ruby_version('2.5.0', option: :max_retries) do
      #     http.max_retries = options[:max_retries]
      #   end
      # end

      # if options[:debug_output]
      #   http.set_debug_output(options[:debug_output])
      #
      #   logger = Logger.new(STDOUT)
      #   http = HTTP.use(logging: {logger: logger})
      # end

      # if options[:ciphers]
      #   http.ciphers = options[:ciphers]
      # end

      # Bind to a specific local address or port
      #
      # @see https://bugs.ruby-lang.org/issues/6617
      # if options[:local_host]
      #   from_ruby_version('2.0.0', option: :local_host) do
      #     http.local_host = options[:local_host]
      #   end
      # end

      # if options[:local_port]
      #   from_ruby_version('2.0.0', option: :local_port) do
      #     http.local_port = options[:local_port]
      #   end
      # end

      http
    end