How to make the msearch query case insensitive?

[yovthxx]

Hello!

I am using msearch with sqlalchemy in my project and found out in testing that it is case sensitive. Is there a viable way to sort it out? I know to make a basic sqlalchemy query case insensitive using lower/upper, but I couldn't find a way to implement the same for your msearch syntax.

Please help :)

edit: Adding my queries as an example. First - search query that I would like to make case insensitive, second one is my solution to the same situation outside of msearch.

1. posts = Posts.query.msearch(keyword, fields=['title', 'description', 'content'], limit=20).limit(10)
2. user = Users.query.filter(func.lower(Users.username) == func.lower(username.data)).first()

[tnedev]

I was wondering the same.

[yovthxx]

Up! Still relevant

[honmaple]

I have added an example about query case insensitive, please check it.

__msearch_schema__ = {
    "title": TEXT(
        stored=True,
        analyzer=RegexTokenizer() | CaseSensitivizer(),
        sortable=False),
    "content": TEXT(
        stored=True,
        analyzer=RegexTokenizer(),
        sortable=False,
    )
}

https://github.com/honmaple/flask-msearch/blob/master/test/test_whoosh.py#L61

[yovthxx]

Thank you for your response. I edited my models file, added imports, schema and analyzers but it didn't work. I don't get any errors, it just works the way it used to and gives me the same result. I left the query and routes the same, because I think the schema is what matters there, right?

[tnedev]

@honmaple It works for me. I had to change to whoosh engine. And update the index on start. But it looks there is no need of adding CaseSentivizer as there are already such filters by whoosh.

However, the search now works only with whole words.

[honmaple]

@yovthxx Can you show the search text and search keywords？

@tnedev Well, now works only with whole words this is because of the selected whoosh analyzer that only support whole word index, if you want support more keywords search, you can use

class CaseSensitivizer(Filter):
    def __call__(self, tokens):
        for t in tokens:
            yield t

            raw_text = t.text
            for i in range(len(raw_text) - 2):
                t.text = raw_text[:3 + i]
                yield t

            text = raw_text.lower()
            if text == t.text:
                continue

            for i in range(len(text) - 2):
                if text[:3 + i] != raw_text[:3 + i]:
                    t.text = text[:3 + i]
                    yield t

But this will make search slower.

[tnedev]

Thanks. I'm adding wildcard during the handling of the request and it works fine.

[yovthxx]

@honmaple This is my searchable model, I have imported the RegexTokenizer and added the CaseSentivizer in models

class Posts(db.Model):
    __tablename__ = 'posts'
    __searchable__ = ['title', 'description', 'content']
    __msearch_schema__ = {
                "title": TEXT(
                    stored=True,
                    analyzer=RegexTokenizer() | CaseSensitivizer(),
                    sortable=False),
                "content": TEXT(
                    stored=True,
                    analyzer=RegexTokenizer(),
                    sortable=False,
                )
            }
    id = db.Column(db.Integer, primary_key=True)
    stage = db.Column(db.Integer, nullable=False, default=0)
    title = db.Column(db.String(250), unique=True, nullable=False)
    description = db.Column(db.String(100), nullable=True)
    link = db.Column(db.String(100), nullable=True)
    logo_file = db.Column(db.String(100), nullable=False, default='default.jpg')
    banner_file = db.Column(db.String(100), nullable=False, default='default.jpg')
    date_posted = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
    content = db.Column(db.Text, nullable=False)

And this is my search route

@posts.route("/search", methods=['GET', 'POST'])
def search():
    searchform = SearchForm()
    posts = []
    if searchform.validate_on_submit():
        keyword = searchform.keyword.data
        posts = Posts.query.msearch(keyword, fields=['title', 'content'], limit=10)
        return render_template("search.html", posts=posts, searchform=searchform)
    return render_template("search.html", posts=posts, searchform=searchform)

My init lines related to search are as follows and no engine set in config (default)

from flask_msearch import Search
search = Search()

It seems like the changes take no effect, so I think I missed something. I also hit recursion when I try to delete/update index. Sorry, my first time with Flask and Python in general