feat: implement filtering of terminal tokens
In order to decrease token usage, we have implemented some basic filtering of non-useful input from the terminal, so that it is not passed the the API.
This feature has a checkbox in the Settings view so that a user may enable or disable it.
We have seen good results in decreasing token usage.
There are also some boilerplate documentation files that slipped in there.
Thank you for taking the time to open this PR @tapvt 🙏
Note to reviewers: we'll need to clean up the documentation in this branch, but let's work with the contributor as much of his ideas in as possible (particularly the regex patterns he's found are good filters)
Hey Id love for this PR to be updated with latest version and now that we have a team we can actually merge it. for now im closing it since its stale. Thank you!
@emshvac - Do you want to catch this branch up?