theJayTea
Support reasoning models like deepseek r1
Support reasoning models like deepseek r1 automatically excluding the ["
Hi! While we can add that, I'm curious if you think it's really required.
Reasoning models have much higher response latencies due to the inference-time compute, and really wouldn't be suited to Writing Tools's grammar correction features. In fact, based on my experiments, reasoning models do not perform better with grammar correction (and sometimes overthink). Summaries are also not something that requires huge reasoning models.
Let me know if there's a specific use case or scenario where you feel reasoning models would make a good difference!
Thanks for the issue and the thoughtful response - I have a use case that would surely benefit from using a reasoning model: I would like to have text proofread, revised and commented upon based on a style guide. Many institutions, companies, organisations have style guides and a lot of time is spent aligning texts with these specific requirements. Especially some of the higher level requirements of such a guide (where it goes beyond right and wrong ways of doing things but makes suggestions or offers several alternative wordings or makes appeals to using a certain style and avoiding another one) might be better taken into account b a reasoning model.
@menelic , thanks for your perspective — that makes a lot of sense, and it's something that didn't strike me. Coming to think of it, I'm sure there'd be other use cases too (code modifications, etc.).
And your use case will be especially propelled once we release customisable buttons and prompts (something @momokrono has very kindly mostly implemented already :D).
Now here's some AWESOME news: Writing Tools happens to already support reasoning models haha!
I went through the DeepSeek R1 documentation, and it uses the same OpenAI API we already support. The best part? There are no reasoning tokens (to exclude) when not using streaming, which is how Writing Tools currently operates.
To use R1, simply open Writing Tools settings and set: Base URL: https://api.deepseek.com/v1 API Model: deepseek-reasoner API Key: your DeepSeek API key
The same easy existing support for R1 should extend to the macOS version too.
Annnd it even supports Gemini 2.0 Flash Thinking just fine! (which does not output reasoning tags unless you ask for it with a new API flag). To use it, simply modify your config.json like so (in the next update I'll add it to the Settings dropdown for Gemini models):
@davidcampos and @menelic have fun!
How about using it with Ollama deepseek r1 distilled models? Does it also offer a streaming option to not send the reasoning tokens?
Do you mean using their OpenAI-compatible APIs or their own?
Right now we don't support ollama-native APIs, I've created a branch to add this feature but it hasn't been merged into main yet.
And in any case, I don't know if ollama offers such an option, I will investigate.
Yes, using OpenAI-compatible APIs of Ollama. I have been using this awesome tool with Ollama for a long time and I can say I am a delighted customer 😍
@davidcampos, could you try using one of those models with Ollama (OpenAI API) + Writing Tools? I'm not sure if Ollama follows the convention of not outputting thinking tokens when not streaming.
If it doesn't, we can add an option to remove these tokens.
And haha, I'm glad to hear you're delighted with it.
@momokrono, I'm really sorry I hadn't yet merged the Ollama implementation. It's currently not automatically mergeable, so I'll work on merging it when I get the time in the coming days. Thank you again for creating that!
No worries! I'll have a look tomorrow, hopefully it's a matter of changing a few lines of code
@davidcampos, could you try using one of those models with Ollama (OpenAI API) + Writing Tools?
I just tried it with the proofread option. Using Ollama with deekseek-r1 32b.
Input:
this is a simple sentence with some mistakes
Output:
<think>
Alright, let's take a look at this query. The user has provided a sentence that needs proofreading: "this is a simple sentence with some mstakes." My task is to correct the grammar and spelling without adding any extra comments.
First, I notice that the sentence starts with a lowercase 't'. In English, sentences should start with an uppercase letter, so I'll change that to 'T'.
Next, looking at the word "mstakes," it's clear that there's a typo here. The intended word is probably "mistakes." So, I'll correct that.
I also check for any other grammatical errors or inconsistencies in the sentence structure. The rest of the sentence seems fine—simple and straightforward with no issues.
Since the user specified to maintain the original text structure and writing style, I make sure not to alter anything else beyond the necessary corrections. Additionally, I ensure that the response is only the corrected text without any additional explanations or comments.
Finally, I review the corrected sentence: "This is a simple sentence with some mistakes." It flows correctly, and all errors have been addressed as per the user's instructions.
</think>
This is a simple sentence with some mistakes.
@davidcampos thanks for testing it. hah, that's a lot of thinking.
I'll add a "For some reasoning models: Remove thinking tags" checkbox in the OpenAI/Ollama model selector that lets you get rid of these in the future.
@theJayTea I fixed the ollama provider and just pushed it to dev. It can be merged automatically into main. After that we could remove the ollama branch altogether.
@theJayTea if after testing the dev branch you see no errors or bugs, yes why not!
Mind that something I haven't tested is the compilation into a windows executable, since I don't have a windows PC to test on... I don't think it won't compile, probably the only thing to do is to add the options.json file into the list of files to bundle with the executable, but other than that it should be fine. And on this note, the custom options are for the moment without a GUI for customization, you have to manually specify them into the options.json file.
@momokrono Ah yep! I reckon it'll work just fine, and I'll of course test it and merge it by our next release this week — along with adding a GUI.
@menelic , thanks for your perspective — that makes a lot of sense, and it's something that didn't strike me. Coming to think of it, I'm sure there'd be other use cases too (code modifications, etc.).
And your use case will be especially propelled once we release customisable buttons and prompts (something @momokrono has very kindly mostly implemented already :D).
Now here's some AWESOME news: Writing Tools happens to already support reasoning models haha!
I went through the DeepSeek R1 documentation, and it uses the same OpenAI API we already support. The best part? There are no reasoning tokens (to exclude) when not using streaming, which is how Writing Tools currently operates.
To use R1, simply open Writing Tools settings and set: Base URL: https://api.deepseek.com/v1 API Model: deepseek-reasoner API Key: your DeepSeek API key
The same easy existing support for R1 should extend to the macOS version too.
Annnd it even supports
Gemini 2.0 Flash Thinkingjust fine! (which does not output reasoning tags unless you ask for it with a new API flag). To use it, simply modify yourconfig.jsonlike so (in the next update I'll add it to the Settings dropdown for Gemini models):
@davidcampos and @menelic have fun!
may I ask where is the config.json file on mac located?
@davidcampos, could you try using one of those models with Ollama (OpenAI API) + Writing Tools? I'm not sure if Ollama follows the convention of not outputting thinking tokens when not streaming.
If it doesn't, we can add an option to remove these tokens.
Just tried and thinking tokens do appear as well if using Ollama's OpenAI API and Deepseek-R1. Would be nice if it can be run locally...
@akuropka thanks for the heads up! I'll add a toggle to remove thinking tokens in the future/next update (it's a bit more nuanced as not all models output the same thinking token delimiters). For now, you can use a reasoning model with Gemini/the official R1 API (instead of an R1-distilled local model).