Error raised by inference API: Input validation error: `inputs` must have less than 1000 tokens. Given: xxx

[kfl-sp]

Hey, i have trouble running this chatbot, everytime i send a question, in the get_conversation_chain, i got this error when using HuggingFaceHub with repo google/flan-t5-xxl :

ValueError: Error raised by inference API: Input validation error: inputs must have less than 1000 tokens. Given: 1115

llm = HuggingFaceHub(repo_id="google/flan-t5-xxl", model_kwargs={"temperature":0.5, "max_length":512})

Could you please look at it please ? Thanks !

[kfl-sp]

Uncaught app exception Traceback (most recent call last): File "/home/__/.local/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script exec(code, module.__dict__) File "/home/__/__/__/ask-multiple-pdfs/app.py", line 105, in <module> main() File "/home/__/__/__/ask-multiple-pdfs/app.py", line 82, in main handle_userinput(user_question) File "/home/__/__/__/ask-multiple-pdfs/app.py", line 56, in handle_userinput response = st.session_state.conversation({'question': user_question}) File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 140, in __call__ raise e File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 134, in __call__ self._call(inputs, run_manager=run_manager) File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/conversational_retrieval/base.py", line 110, in _call answer = self.combine_docs_chain.run( File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 239, in run return self(kwargs, callbacks=callbacks)[self.output_keys[0]] File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 140, in __call__ raise e File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 134, in __call__ self._call(inputs, run_manager=run_manager) File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/combine_documents/base.py", line 84, in _call output, extra_return_dict = self.combine_docs( File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/combine_documents/stuff.py", line 87, in combine_docs return self.llm_chain.predict(callbacks=callbacks, **inputs), {} File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/llm.py", line 213, in predict return self(kwargs, callbacks=callbacks)[self.output_key] File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 140, in __call__ raise e File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 134, in __call__ self._call(inputs, run_manager=run_manager) File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/llm.py", line 69, in _call response = self.generate([inputs], run_manager=run_manager) File "/home/__/.local/lib/python3.10/site-packages/langchain/chains/llm.py", line 79, in generate return self.llm.generate_prompt( File "/home/__/.local/lib/python3.10/site-packages/langchain/llms/base.py", line 134, in generate_prompt return self.generate(prompt_strings, stop=stop, callbacks=callbacks) File "/home/__/.local/lib/python3.10/site-packages/langchain/llms/base.py", line 191, in generate raise e File "/home/__/.local/lib/python3.10/site-packages/langchain/llms/base.py", line 185, in generate self._generate(prompts, stop=stop, run_manager=run_manager) File "/home/__/.local/lib/python3.10/site-packages/langchain/llms/base.py", line 436, in _generate self._call(prompt, stop=stop, run_manager=run_manager) File "/home/__/.local/lib/python3.10/site-packages/langchain/llms/huggingface_hub.py", line 112, in _call raise ValueError(f"Error raised by inference API: {response['error']}")

[RobertoDonPedro]

Same issue here. Processing: Created a chunk of size 1110, which is longer than the specified 1000 Asking: ValueError: Error raised by inference API: Input validation error: inputs must have less than 1000 tokens. Given: 1109

[HagaiHen]

Modify on get_text_chunks function the chunk_size and chunk_overlap. It should look like this:

def get_text_chunks(text):
    text_splitter = CharacterTextSplitter(
        separator="\n",
        chunk_size=300,
        chunk_overlap=50,
        length_function=len
    )
    chunks = text_splitter.split_text(text)
    return chunks

[RobertoDonPedro]

For the processing part I managed to run it by replacing the CharacterTextSplitter with RecursiveCharacterTextSplitter as follows: from langchain.text_splitter import RecursiveCharacterTextSplitter and text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200) However, the real problem appears to be that the remote inference api is limited...

[maotoledo]

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)

Does not work for me.

[shayons]

Same issue. RecursiveCharacterTextSplitter is not working for me. Any pointers on how to fix?

[meang123]

def get_text_chunks(text): text_splitter = RecursiveCharacterTextSplitter( chunk_size=400, chunk_overlap=50, length_function=len ) chunks = text_splitter.split_documents(text) return chunks

use a default separators , then it's works