Support for Ollama gpt-oss:20b

[alexsokolow]

Description:
When running LangExtract with gpt-oss:20b in Ollama, the extraction fails. It seems that the output format is incompatible with LangExtract’s expectations.

Reproduction Steps:

python quickstart.py --model-id gpt-oss:20b

🚀 Running Ollama quick-start example with gpt-oss:20b...
--------------------------------------------------
LangExtract: Processing, current=51 chars, processed=51 chars:  [00:00]ERROR:absl:Failed to parse content.
Traceback (most recent call last):
  File "/Users/-/code/langextract/langextract_venv/lib/python3.11/site-packages/langextract/resolver.py", line 349, in _extract_and_parse_content
    parsed_data = json.loads(content)
...
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 5 (char 4)
LangExtract: Processing, current=51 chars, processed=51 chars:  [00:02]

Error: ResolverParsingError: Failed to parse content.

[wc-lei]

same problem here.

[paulbei]

same here

[nathansutton]

same here.

[Smonkey123]

Note that when deploying a local LLM using ollama, your local LLM needs to be parameterized to support JSON output, so that it can be parsed by the JSON parser of lx. If the output is not in JSON format, it will indicate a parsing failure. It is also worth noting that for some models, you can instruct them to output in JSON format in the prompt. However, this method is not 100% effective and thus may cause some parsing failures. A better solution is to switch to a model supported by Ollama that can output in JSON format.

[aksg87]

Thanks for reporting this issue.

Did a quick test and it seems to be an integration problem between Ollama and gpt-oss:20b specifically, though I'm not completely sure.

Here's what I found:

import requests
import json

# gemma2:2b - works fine
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "gemma2:2b",
        "prompt": "What is 2+2? Reply with JSON containing the answer.",
        "format": "json",
        "stream": False,
    }
)
print(f"gemma2:2b: {response.json()['response']}")
# Output: {"answer": 4}

# gpt-oss:20b - doesn't output JSON
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "gpt-oss:20b",
        "prompt": "What is 2+2? Reply with JSON containing the answer.",
        "format": "json",
        "stream": False,
    }
)
print(f"gpt-oss:20b: {response.json()['response']}")
# Output: (empty or non-JSON text)

It looks like gpt-oss:20b isn't respecting Ollama's format: "json" parameter, while other models like gemma2:2b, llama3.2, and mistral work fine with the same code.

Please let me know if you find an issue with LangExtract's connection to Ollama itself.

[paulbei]

I think the issue is the new OpenAI Harmony Response Format. Could that be the case? https://cookbook.openai.com/articles/openai-harmony

[acer2265848]

I using gpt-oss-20b with LM studio check the code below, it's work

import langextract as lx
from langextract.providers.openai import OpenAILanguageModel

class CustomOpenAIModel(OpenAILanguageModel):
    def _process_single_prompt(self, prompt, config):

        api_params = {
            "model": self.model_id,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.1,
            "max_tokens": 2000,
        }
        
        try:
            response = self._client.chat.completions.create(**api_params)
            output_text = response.choices[0].message.content
            return lx.inference.ScoredOutput(score=1.0, output=output_text)
        except Exception as e:
            raise lx.exceptions.InferenceRuntimeError(
                f'Custom OpenAI API error: {str(e)}', original=e
            ) from e



custom_model = CustomOpenAIModel(
    model_id="gpt-oss-20b",
    api_key="dummy",
    base_url="http://127.0.0.1:1234/v1"
)

result = lx.extract(
    text_or_documents=input_text,
    prompt_description=prompt,
    examples=examples,
    model=custom_model,
    fence_output=False,
    use_schema_constraints=False
)

[nikmd1306]

Thank you bro!!