sd-scripts Random crashes: 'NoneType' object has no attribute 'ndim'

Training is intermittently crashing on some sample generations. Error:

[rank3]: Traceback (most recent call last):
[rank3]: File "/root/run/kohya/flux_train.py", line 905, in
[rank3]: train(args)
[rank3]: File "/root/run/kohya/flux_train.py", line 760, in train
[rank3]: flux_train_utils.sample_images(
[rank3]: File "/root/run/kohya/library/flux_train_utils.py", line 113, in sample_images
[rank3]: sample_image_inference(
[rank3]: File "/root/run/kohya/library/flux_train_utils.py", line 229, in sample_image_inference
[rank3]: x = denoise(flux, noise, img_ids, t5_out, txt_ids, l_pooled, timesteps=timesteps, guidance=scale, t5_attn_mask=t5_attn_mask)
[rank3]: File "/root/run/kohya/library/flux_train_utils.py", line 314, in denoise
[rank3]: pred = model(
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward
[rank3]: return model_forward(*args, **kwargs)
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/accelerate/utils/operations.py", line 807, in call
[rank3]: return convert_to_fp32(self.model_forward(*args, **kwargs))
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 43, in decorate_autocast
[rank3]: return func(*args, **kwargs)
[rank3]: File "/root/run/kohya/library/flux_models.py", line 1004, in forward
[rank3]: if img.ndim != 3 or txt.ndim != 3:
[rank3]: AttributeError: 'NoneType' object has no attribute 'ndim'

Sep 08 '24 15:09 ipb26

Hello @cnarte! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this issue together!

I found a similar issue where users encountered a TypeError: Cannot read properties of undefined (reading 'text') when using ChatGoogleGenerativeAI with tools. This error typically occurs when the response from the Google Generative AI API does not contain the expected structure, specifically the text property.

Solution

To address this issue, you should ensure that the response from the API is correctly structured and contains the text property. Additionally, you can add checks to handle cases where the text property might be missing.

Here is a potential fix in the _generate method to handle undefined properties:

async _generate(
  messages: BaseMessage[],
  options: this["ParsedCallOptions"],
  runManager?: CallbackManagerForLLMRun
): Promise<ChatResult> {
  const prompt = convertBaseMessagesToContent(
    messages,
    this._isMultimodalModel
  );
  const parameters = this.invocationParams(options);

  // Handle streaming
  if (this.streaming) {
    const tokenUsage: TokenUsage = {};
    const stream = this._streamResponseChunks(messages, options, runManager);
    const finalChunks: Record<number, ChatGenerationChunk> = {};

    for await (const chunk of stream) {
      if (chunk.text) {
        // Process the chunk
      } else {
        console.error("Received chunk without text property:", chunk);
      }
    }
  } else {
    const response = await this.client.generateContent({
      contents: [prompt],
      ...parameters,
    });

    if (response && response.text) {
      return mapGenerateContentResultToChatResult(response);
    } else {
      throw new Error("Response does not contain text property");
    }
  }
}

This code adds checks to ensure that the text property is present before attempting to access it, which should prevent the TypeError from occurring ^[1].

Additional Steps

Ensure Correct Dependencies: Make sure you have the necessary dependencies installed and configured correctly:
```
npm install @langchain/google-genai @langchain/core
```

Check package.json: Ensure your package.json includes the correct dependencies and resolutions:

{
  "name": "your-project",
  "version": "0.0.0",
  "dependencies": {
    "@langchain/core": "^0.3.0",
    "@langchain/google-genai": "^0.0.0"
  },
  "resolutions": {
    "@langchain/core": "^0.3.0"
  },
  "overrides": {
    "@langchain/core": "^0.3.0"
  },
  "pnpm": {
    "overrides": {
      "@langchain/core": "^0.3.0"
    }
  }
}

Set Up Environment Variable:
```
export GOOGLE_API_KEY=your-api-key
```

Initialize ChatGoogleGenerativeAI Correctly:

import { ChatGoogleGenerativeAI } from "@langchain/google-genai";

const model = new ChatGoogleGenerativeAI({
  modelName: "gemini-pro",
  maxOutputTokens: 2048,
});
const response = await model.invoke(new HumanMessage("Hello world!"));

By following these steps, you should be able to avoid the TypeError and ensure that your integration with the Google Generative AI API works correctly ^[1]^[2].

To continue talking to Dosu, mention @dosu.

Sep 17 '24 06:09 dosubot[bot]

But how can I edit the Package implementation?

Sep 17 '24 06:09 cnarte

To edit the package implementation, you can follow these steps:

Extract the Package: Use the yarn patch command to extract the package to a temporary directory.
Make Your Changes: Edit the files in the extracted directory as needed.
Commit Your Changes: Use the yarn patch-commit command to commit your changes and create a patch file.

Here is a code snippet to guide you through the process:

# Extract the package
yarn patch @langchain/google-genai

# Make your changes in the extracted directory
# (The directory path will be displayed in the terminal output)

# Commit your changes
yarn patch-commit -s <path-to-extracted-directory>

This will store a patch file based on your changes, which you can then apply to your project ^[1].

To continue talking to Dosu, mention @dosu.

Sep 17 '24 06:09 dosubot[bot]

@dosu that wont solve problem just throw a new error.

Sep 17 '24 11:09 cnarte

To resolve the issue with TypeError: Cannot read properties of undefined (reading 'text') when using ChatGoogleGenerativeAI with tools in Langraph, ensure that the properties you are trying to access are correctly defined and available. In the provided code, the FakeBrowserTool class and its schema are correctly set up, but you should verify that the response structure from modelWithTools.invoke is as expected.

Here is a revised version of the code with additional checks to ensure that the properties are correctly accessed:

import { StructuredTool } from "@langchain/core/tools";
import { ChatGoogleGenerativeAI } from "@langchain/google-genai";
import { z } from "zod";

const model = new ChatGoogleGenerativeAI({
  model: "gemini-pro",
});

// Define your tool
class FakeBrowserTool extends StructuredTool {
  schema = z.object({
    url: z.string(),
    query: z.string().optional(),
  });

  name = "fake_browser_tool";

  description =
    "useful for when you need to find something on the web or summarize a webpage.";

  async _call(_: z.infer<this["schema"]>): Promise<string> {
    return "fake_browser_tool";
  }
}
const tool = new FakeBrowserTool();

// Bind your tools to the model
const modelWithTools = model.withStructuredOutput(tool.schema, {
  name: tool.name, // this is optional
});
// Optionally, you can pass just a Zod schema, or JSONified Zod schema
// const modelWithTools = model.withStructuredOutput(
//   zodSchema,
// );

const res = await modelWithTools.invoke([
  [
    "human",
    "Search the web and tell me what the weather will be like tonight in new york. use a popular weather website",
  ],
]);

if (res && res.url && res.query) {
  console.log(res);
} else {
  console.error("Unexpected response structure:", res);
}

/*
{
  url: 'https://www.accuweather.com/en/us/new-york-ny/10007/night-weather-forecast/349014',
  query: 'weather tonight'
}
*/

This code includes a check to ensure that res, res.url, and res.query are defined before attempting to access them. This should help prevent the TypeError you are encountering.

Additionally, make sure you have installed the @langchain/google-genai package as required:

npm install @langchain/google-genai

Refer to the LangChain documentation for more details on using the ChatGoogleGenerativeAI class and its methods ^[1]^[2]^[3]^[4].

To continue talking to Dosu, mention @dosu.

Sep 17 '24 11:09 dosubot[bot]

I'd need to dig into the @langchain/google-genai package to be sure, but my guess is that this isn't caused by tools specifically but that a finishReason of "SAFETY" or something else is being triggered. In these cases, there is no text returned, so trying to access the text element (which is what is done internally) is triggering the exception.

A slightly biased suggestion, but you may wish to switch to using the @langchain/google-gauth or @langchain/google-webauth packages instead. These don't avoid the issue, but they do let you manage it a bit better with a SecurityManager. You'll then be using the ChatGoogle class.

Make sure you are using version 0.1.0 of either and version 0.1.1 of @langchain/google-common. You'll also need to explicitly depend on @langchain/common version 0.3.0.

Sep 17 '24 13:09 afirstenberg

@cnarte can you provide some code I can use to reproduce this error?

Sep 17 '24 17:09 bracesproul

Hi, @cnarte. I'm Dosu, and I'm helping the LangChain JS team manage their backlog. I'm marking this issue as stale.

Issue Summary:

You encountered a TypeError related to an undefined 'text' property when using the Google Generative AI package with Langraph.
I suggested checking the API response structure and provided a code snippet to handle missing properties, but you reported a new error.
@afirstenberg mentioned the issue might be due to a finishReason like "SAFETY" and suggested using @langchain/google-gauth.
@bracesproul requested a code example to reproduce the error for further investigation.

Next Steps:

Please confirm if this issue is still relevant with the latest version of LangChain JS. If it is, you can keep the discussion open by commenting here.
If there is no further activity, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!

Dec 17 '24 16:12 dosubot[bot]