Assistant Timeout

[james-haddock]

Check for existing issues

• [X] Completed

Describe the bug / provide steps to reproduce it

With the Zed Assistant, I like using a local LLM with a relatively large context size. This can sometimes make response times quite slow. Since updating to 0.128.3, I have been receiving a timeout message if it doesn't receive a response quickly enough: "Request or operations took longer than the configured timeout time." Is there a way to configure the timeout time to be longer?

Environment

Zed: v0.128.3 (Zed) OS: macOS 14.4.0 Memory: 32 GiB Architecture: aarch64

If applicable, add mockups / screenshots to help explain present your vision of the feature

If applicable, attach your ~/Library/Logs/Zed/Zed.log file to this issue.

2024-03-28T12:12:11+00:00 [WARN] request completed with error: request or operation took longer than the configured timeout time

[craigcomstock]

Trying to get a local LLM working with assistant with ollama I can't use the LLM due to a timeout.

Is there a configuration item somewhere I can change?

[craigcomstock]

Looking at the code a bit it would seem that the issue might be a timeout in the http client?

From zed/crates/assistant/src/completion_provider/open_ai.rs

        async move {
            let api_key = api_key.ok_or_else(|| anyhow!("missing api key"))?;
            let request = stream_completion(http_client.as_ref(), &api_url, &api_key, request);
            let response = request.await?;
            let stream = response
                .filter_map(|response| async move {
                    match response {
                        Ok(mut response) => Some(Ok(response.choices.pop()?.delta.content?)),
                        Err(error) => Some(Err(error)),
                    }
                })
                .boxed();
            Ok(stream)
        }

and stream_completion is from

use open_ai::{stream_completion, Request, RequestMessage, Role as OpenAiRole};

stream_completion seems to be implemented in zed/crates/open_ai/src/open_ai.rs.

use util::http::{AsyncBody, HttpClient, Method, Request as HttpRequest};

Which is in zed/crates/util/src/http.rs

maybe this is the timeout?

pub fn client() -> Arc<dyn HttpClient> {
    Arc::new(
        isahc::HttpClient::builder()
            .connect_timeout(Duration::from_secs(5))
            .low_speed_timeout(100, Duration::from_secs(5))
            .proxy(http_proxy_from_env())
            .build()
            .unwrap(),
    )
}

So this is using

pub use isahc::{
    http::{Method, StatusCode, Uri},
    AsyncBody, Error, HttpClient as IsahcHttpClient, Request, Response,
};

https://docs.rs/isahc/latest/isahc/

I see https://docs.rs/isahc/latest/isahc/config/trait.Configurable.html#method.timeout

and

use isahc::{prelude::*, Request};
use std::time::Duration;

// This page is too slow and won't respond in time.
let response = Request::get("https://httpbin.org/delay/10")
    .timeout(Duration::from_secs(5))
    .body(())?
    .send()
    .expect_err("page should time out");

I will see if I can build zed locally and hack on this a bit. :)

[james-haddock]

Looking at the code a bit it would seem that the issue might be a timeout in the http client?

From zed/crates/assistant/src/completion_provider/open_ai.rs

        async move {
            let api_key = api_key.ok_or_else(|| anyhow!("missing api key"))?;
            let request = stream_completion(http_client.as_ref(), &api_url, &api_key, request);
            let response = request.await?;
            let stream = response
                .filter_map(|response| async move {
                    match response {
                        Ok(mut response) => Some(Ok(response.choices.pop()?.delta.content?)),
                        Err(error) => Some(Err(error)),
                    }
                })
                .boxed();
            Ok(stream)
        }

and stream_completion is from

use open_ai::{stream_completion, Request, RequestMessage, Role as OpenAiRole};

stream_completion seems to be implemented in zed/crates/open_ai/src/open_ai.rs.

use util::http::{AsyncBody, HttpClient, Method, Request as HttpRequest};

Which is in zed/crates/util/src/http.rs

maybe this is the timeout?

pub fn client() -> Arc<dyn HttpClient> {
    Arc::new(
        isahc::HttpClient::builder()
            .connect_timeout(Duration::from_secs(5))
            .low_speed_timeout(100, Duration::from_secs(5))
            .proxy(http_proxy_from_env())
            .build()
            .unwrap(),
    )
}

So this is using

pub use isahc::{
    http::{Method, StatusCode, Uri},
    AsyncBody, Error, HttpClient as IsahcHttpClient, Request, Response,
};

https://docs.rs/isahc/latest/isahc/

I see https://docs.rs/isahc/latest/isahc/config/trait.Configurable.html#method.timeout

and

use isahc::{prelude::*, Request};
use std::time::Duration;

// This page is too slow and won't respond in time.
let response = Request::get("https://httpbin.org/delay/10")
    .timeout(Duration::from_secs(5))
    .body(())?
    .send()
    .expect_err("page should time out");

I will see if I can build zed locally and hack on this a bit. :)

Thanks!

I've rolled back to 0.127.5 for now, seems to be functioning fine again!

[craigcomstock]

@james-haddock I have a hack branch/PR up and the assistant now works for me with a local ollama mistral model. I will work on (or maybe someone else like you will) fixing this up properly. Maybe add a timeout or -1 (for no timeout) to the assistant settings?

[JosephTLyons]

At the moment, I dont thin we have a setting for this, but this could be a pretty simple addition, i think.

[craigcomstock]

At the moment, I dont thin we have a setting for this, but this could be a pretty simple addition, i think.

I will try to make this change. Hold my beer. 👍

[craigcomstock]

@JosephTLyons as this timeout setting would likely be optional don't you think it could be added to the V1 settings that are already there? It won't break compatibility.

[MatheusMartins]

While trying out Ollama, I faced this same issue on models llama2 and mistral. I managed to work around it by using tinydolphin, which is a 1.1B Parameter token, replying much faster, hence bypassing the timeout issue. In case anyone is also looking for a solution while we can't set a custom timeout, this is how I did it:

ollama pull tinydolphin # download smaller model
ollama cp tinydolphin gpt-4-turbo-preview  # copy to gpt-4-turbo-preview to be accepted by Zed

Zed Settings:

    "provider": {
      "name": "openai",
      "api_url": "http://localhost:11434/v1"
    }

My example: how to change the page title?

To change the page title, you can modify it in two ways:

1. Within your JavaScript code (i.e., within the main.tsx), use document.title API to update the page title.

document.title = "My New Title";

2. In your HTML template file (/index.html, /* src */ or any other), replace the <title> element with the desired text. This method ensures the updated title is displayed when you load your page and throughout the website, so it's generally more reliable.

However, changing the page title externally (i.e., by clicking a link or changing URL) could require advanced knowledge of JavaScript and can also impact the browser tab state and may cause unpredictable issues on different browsers and platforms. In general, when making changes to the HTML/CSS layouts for a web app's pages, using server-side logic is recommended to prevent browser inconsistencies.

Not mindblown, but I think I had never written a document.title line before, so learned something new 🙃 Thank you Zed community!

PS: -Read the contributing doc, but not a hardcore github collaborator - apolgies if I missed on any guidelines, please call out. -Hardware: MacBook Pro M1 - 16GB

[ArmaanKatyal]

@craigcomstock, are you still working on this issue? I see a draft PR up. If not, I can work on the issue.

[craigcomstock]

@craigcomstock, are you still working on this issue? I see a draft PR up. If not, I can work on the issue.

That would be great if you could take it over. I thought I would have some time but now do not.

[maxdeviant]

I just merged https://github.com/zed-industries/zed/pull/11668, if anyone wants to build from main and try it out to see if it resolves their issues.

[JosephTLyons]

This is now available in Zed v0.136.0-pre.

[tdomhan]

The PR mentions only OpenAI. How to set a timeout for ollama?

[bennetbo]

@tdomhan You should be able to add a timeout like this:

"language_models": {
    "ollama": {
      "low_speed_timeout_in_seconds": 30
    }
}

[XiaoConstantine]

@bennetbo I m using gemini but seems assistant is not respecting value I set here:

"language_models": {
    "google": {
      "low_speed_timeout_in_seconds": 60
    }

}

any suggestions? here's my setting.json

{
  "assistant": {
    "default_model": {
      "provider": "google",
      "model": "gemini-1.5-flash"
    },

    "version": "2"
  },
  "language_models": {
    "google": {
      "low_speed_timeout_in_seconds": 60
    }
  },
  ...

zed version:

❯ zed --version
Zed 0.151.1 – /Applications/Zed.app

[bennetbo]

@XiaoConstantine Looks like you're correct and we are actually not respecting the low_speed_timeout_in_seconds setting for the google AI provider. I just put up #17423 which should fix the issue.

[XiaoConstantine]

Thanks @bennetbo for the quick fix, I have tried build from master and seems the timeout error is indeed gone.