JSON Decode Error with Postman

[himansharma21]

Describe the bug

JSONDecodeError when sending API request through Postman.

It is happening at the line params = json.loads(string) within modules/api.py. The string has single quotes instead of double quotes required to decode the JSON.

I have been able to fix it on my machine by replacing params = json.loads(string) with params = ast.literal_eval(string) . Let me know if anyone has encountered this, it should be pretty simple fix.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Sending a request to the API using Postman.

Screenshot

No response

Logs

Traceback (most recent call last):
  File "C:\Users\username\anaconda3\envs\tgwui\lib\site-packages\gradio\routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\Users\username\anaconda3\envs\tgwui\lib\site-packages\gradio\blocks.py", line 1108, in process_api
    result = await self.call_function(
  File "C:\Users\username\anaconda3\envs\tgwui\lib\site-packages\gradio\blocks.py", line 929, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Users\username\anaconda3\envs\tgwui\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\username\anaconda3\envs\tgwui\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\Users\username\anaconda3\envs\tgwui\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\Users\username\anaconda3\envs\tgwui\lib\site-packages\gradio\utils.py", line 490, in async_iteration
    return next(iterator)
  File "F:\text-generation-webui\modules\api.py", line 30, in generate_reply_wrapper
    params = json.loads(string)
  File "C:\Users\username\anaconda3\envs\tgwui\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\username\anaconda3\envs\tgwui\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\username\anaconda3\envs\tgwui\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)

### System Info

```shell
Windows 10, RTX 3060ti, Ryzen 5 3600

[extramagichours]

Yes, I believe this could be issue I've been having

That being said, now I'm point at 127.0.0.1:5000/api/v1/generate which I understand is the Kobold API. Is the Gradio API different? Before I was using: 127.0.0.1:7860/run/textgen with no issues and very reliable results.

Thanks for digging into this one!

[himansharma21]

I do know that the main API has been changed sometime recently. It seems to still work for the api-example.py but does not work for some applications.

If the fix works for you, maybe I should create a PR?

[extramagichours]

Sorry, not sure to be honest. I tried implementing your changes (you also need to import ast but I get a 404 response form the API server at :7860/run/textgen

Let me know if there's something else I can check to see if this fix worked. Maybe give me an example POST request you're sending just to be sure it's not me?

[himansharma21]

Point the request to http://127.0.0.1:7860/run/textgen. I am not sure which parameters you're using in your request, I am getting mine from api-example.py

[extramagichours]

I'm just getting a 404. I guess my issue is not related. What's so strange is just going back a previous commit before the API changed a few weeks ago just works... but with updates the API at http://127.0.0.1:7860/run/textgen is just unreachable for me...

[himansharma21]

Yeah your issue seems unrelated. Create an issue if you have not already, make sure you share your params with which you're starting server.py.

[extramagichours]

Sorry - let me refocus here. Your fix does work. My issue was unrelated, but this just restored functionality for me at http://127.0.0.1:7860/run/textgen

I had to use this format for my request though:

{ 
    "data": [[ 
    "What I would like to say is the following:", { 
            "max_new_tokens": 200, 
            "do_sample": true, 
            "temperature": 0.5, 
            "top_p":0.9, 
            "typical_p":-1, 
            "repetition penalty": 1.05, 
            "encoder_repetition_penalty": 1.0, 
            "top_k": 0, "min_length": 0, 
            "no_repeat_ngram_size": 0, 
            "num_beams": -1, 
            "penalty_alpha": 0, 
            "length_penalty":-1, 
            "early_stopping": false, 
            "seed": -1,
            "add_bos_token":false,
            "truncation_length":2000,
            "custom_stopping_strings":[],
            "ban_eos_token":false
        } 
    ]] 
}

Apologies to have added confusion.

[himansharma21]

Yeah the format had changed recently. Glad to have helped partially.

[Khyretos]

i cannot make it work with none of the jsons from the previous comments, am i missing something?

https://user-images.githubusercontent.com/25864917/232466341-2c0bced3-6f24-4f58-9066-15b71faa42d3.mp4

[Khyretos]

Never mind im stupid, i just saw the api example and did not place the -listen --no-stream flags... my bad

[hydrix9]

was getting this error too. lean into the curve gentlemen, lean into the curve

ChatGPT4:

It appears that the generate_reply_wrapper function expects a single string as input, which should contain a JSON-encoded list with the prompt and the parameters. However, the JSON data you were sending through Postman had the prompt and parameters as separate elements within an array under the "data" key.

To make the server-side code compatible with the JSON data you are sending, you need to modify the api.py file. First, update the generate_reply_wrapper function to accept two separate arguments: prompt and params. Then, remove the line that loads the JSON data, as the data is already in the correct format:

def generate_reply_wrapper(prompt, params):
    # Provide defaults so as to not break the API on the client side when new parameters are added
    # ...

    generate_params.update(params)
    stopping_strings = generate_params.pop('stopping_strings')
    for i in generate_reply(prompt, generate_params, stopping_strings=stopping_strings):
        yield i

Next, modify the create_apis() function to define the input components for the prompt and params. Replace the t1 and t2 Textboxes with two new components: a Textbox for the prompt and a JSON component for the params

def create_apis():
    prompt_input = gr.Textbox(visible=VISIBLE)
    params_input = gr.JSON(visible=VISIBLE)
    dummy = gr.Button(visible=VISIBLE)

    input_params = [prompt_input, params_input]
    output_params = [shared.gradio[k] for k in ['markdown', 'html']]
    dummy.click(generate_reply_wrapper, input_params, output_params, api_name='textgen')

Now, the server-side code should be compatible with the JSON data you are sending through Postman. You can send the data in the following format:

{
  "data": [
    "What I would like to say is the following: ",
    {
      "max_new_tokens": 200,
      "do_sample": true,
      "temperature": 0.72,
      "top_p": 0.73,
      "typical_p": 1,
      "repetition_penalty": 1.1,
      "encoder_repetition_penalty": 1.0,
      "top_k": 0,
      "min_length": 0,
      "no_repeat_ngram_size": 0,
      "num_beams": 1,
      "penalty_alpha": 0,
      "length_penalty": 1,
      "early_stopping": false,
      "seed": -1,
      "add_bos_token": true,
      "truncation_length": 2048,
      "ban_eos_token": false,
      "skip_special_tokens": true,
      "stopping_strings": []
    }
  ]
}

Make these changes in your api.py file, and the issue should be resolved.

"how can I change the postman data to work with the current api.py?"

To modify the Postman data to work with the current api.py, you need to combine the prompt and the parameters into a single JSON-encoded string, which will be passed as the input to the generate_reply_wrapper function.

{
    "data": [
        "[\"What I would like to say is the following: \", {\"max_new_tokens\": 200, \"do_sample\": true, \"temperature\": 0.72, \"top_p\": 0.73, \"typical_p\": 1, \"repetition_penalty\": 1.1, \"encoder_repetition_penalty\": 1.0, \"top_k\": 0, \"min_length\": 0, \"no_repeat_ngram_size\": 0, \"num_beams\": 1, \"penalty_alpha\": 0, \"length_penalty\": 1, \"early_stopping\": false, \"seed\": -1, \"add_bos_token\": true, \"truncation_length\": 2048, \"ban_eos_token\": false, \"skip_special_tokens\": true, \"stopping_strings\": []}]"
    ]
}

This JSON object contains a single string, which is a JSON-encoded list. The list has two elements: the prompt and the parameters dictionary.

Make sure to send this JSON data with the "Content-Type" header set to "application/json". This should work with the current implementation of api.py.

hope this helps