Getting 'undefined' response in nodejs implementation of the sdk

[painful-bug]

trafficstars

when I am running this code :

const h = require('@huggingface/inference')

const Hf = new h.HfInference(process.env.HUGGINGFACE_API_KEY)


// Request the HuggingFace API for the response
async function start() {
  const response = await Hf.textGenerationStream({
    model: 'oasst-sft-4-pythia-12b-epoch-3.5',
    inputs: "Hello",
    parameters: {
      max_new_tokens: 200,
      // @ts-expect-error -- The HfInference API doesn't have types for `typical_p` even though it's necessary
      typical_p: 0.2,
      repetition_penalty: 1,
      truncate: 1000,
    },
  })

  const completion = response.generated_text
  console.log(completion)
}

start()

The output is undefined. What is wrong here?

[shuding]

Can you check what response is? There might be some error message attached.

[karlbateman]

Running the following:

const h = require('@huggingface/inference');

const Hf = new h.HfInference(process.env.HUGGINGFACE_API_KEY);

// Request the HuggingFace API for the response
async function start() {
  const response = Hf.textGenerationStream({
    model: 'oasst-sft-4-pythia-12b-epoch-3.5',
    inputs: 'Hello',
    parameters: {
      max_new_tokens: 200,
      typical_p: 0.2,
      repetition_penalty: 1,
      truncate: 1000,
    },
  });

  console.log('Response: ', response);
  const completion = response.generated_text;
  console.log('Completion: ', completion);
}

start();

Outputs:

Response:  Object [AsyncGenerator] {}
Completion:  undefined

[shuding]

As you can see, response.generated_text is not defined. Where did you find the code of response.generated_text in the docs?

We have an example here here that shows how the SDK should be used:

[karlbateman]

I'm trying to debug an issue with the example you've shared which is documented. At first my implementation was working but after a few interactions I am no longer getting results.

My investigation brought me to this issue in which I'd thought the source was with the API reponse, however I have been unable to debug where the problem occurs.

First thought was perhaps there is an issue with the streams not being closed which is exacerbated when NextJS restarts the dev server on code change. I'm currently unable to confirm this and as there are so many moving parts, I fear this may take some time to debug.

In the meantime, I have fallen back to using the OpenAI integration which has been easier to setup.

Absolutely love the project and the work that has gone into streamlining the integrations ❤️ I'm excited to see where this project goes.

[shuding]

Thank you! Feel free to open an issue or discussion whenever you're running into a problem again. I'll close this for now as there's nothing to track at the moment.

[hypnocill]

@karlbateman did you find what was the issue? I encounter the same situation in which it just stops working and I can't error handle it. Code change hot reloading came to my mind too but can't confirm it