[I] Note list panel issue when searching cyrillic words in different case

[ysaroka]

Expected behaviour

Cyrillic words must be searched in the same mode as Latin, in case-insensitive mode.

Actual behaviour

Cyrillic words are searched only in case-sensitive mode, while Latin words are searched case-insensitively (this is correct behavior)

I did not find the ability to configure the search engine in any way. This is probably a issue.

Steps to reproduce

Video (Tested words: "установка", "усТанОвка"): https://www.youtube.com/watch?v=jP_tyzJ1pkc

Information about the application, settings and environment

Expand

QOwnNotes Debug Information

General Info

Current Date: Чт янв 16 06:03:01 2020 Version: 20.1.9 Build date: Jan 15 2020 Build number: 5275 Platform: windows Operating System: Windows 10 (10.0) Build architecture: i386 Current architecture: x86_64 Release: AppVeyor Qt Version (build): 5.13.2 Qt Version (runtime): 5.13.2 Portable mode: yes Settings path / key: D:/apps/qownnotes/Data/PBE/QOwnNotes.ini Application database path: D:\apps\qownnotes\Data\QOwnNotes.sqlite Application arguments: D:\apps\qownnotes\QOwnNotes.exe, --portable Qt Debug: 0 Locale (system): ru_RU Locale (interface): en Primary screen resolution: 1366x768 Icon theme: breeze-qownnotes Notes in current note folder: 2 Calendar items: 0 Enabled scripts: 0

Server Info

serverUrl: empty appIsValid: no notesPathExists: empty connectionErrorMessage: empty

Spellchecking

Enabled: false Selected language: empty Language codes: empty Language names: empty Application dictionaries path: D:\apps\qownnotes\Data\dicts

Note folders

currentNoteFolderId: 1

Note folder по умолчанию

id: 1 isCurrent: yes activeTagId: 0 localPath: D:\apps\qownnotes\Data\..\..\..\Dropbox\docs\!notes\qownnotes-notes remotePath: Notes cloudConnectionId: 1 isShowSubfolders: yes isUseGit: no activeNoteSubFolder name: Temp database file: D:\apps\qownnotes\Data\..\..\..\Dropbox\docs\!notes\qownnotes-notes\notes.sqlite

Cloud connections

Cloud connection По умолчанию

id: 1 isCurrent: yes serverUrl: empty username: empty

Enabled scripts

Settings

AboutDialog/geometry (QByteArray): <binary data> ActiveNoteHistoryItem (NoteHistoryItem): <binary data> Debug/fakeOldVersionNumber (bool): false Debug/fileLogging (bool): false DistractionFreeMode/editorWidthMode (QString): 1 Editor/CurrentSchemaKey (QString): EditorColorSchema-6033d61b-cb96-46d5-a3a8-20d5172017eb Editor/autoBracketClosing (bool): true Editor/autoBracketRemoval (bool): true Editor/centerCursor (QString): false Editor/disableCursorBlinking (bool): false Editor/editorWidthInDFMOnly (bool): true Editor/highlightCurrentLine (bool): true Editor/indentSize (int): 4 Editor/useTabIndent (bool): false Editor/vimMode (bool): false FileDialog/LastPath (QString): D:/apps/qownnotes/!settings FileDialog/LastPath-ExternalEditor (QString): D:/apps/SublimeTextPortable FileDialog/LastPath-SettingsExport (QString): D:/apps/qownnotes/!settings IssueAssistantDialog/geometry (QByteArray): <binary data> LastUpdateCheck (QDateTime): 2020-01-16T05:44:45.912 LinkDialog/geometry (QByteArray): <binary data> LocalTrashDialog/geometry (QByteArray): <binary data> LogWidget/criticalLog (bool): true LogWidget/debugLog (bool): true LogWidget/fatalLog (bool): true LogWidget/infoLog (bool): true LogWidget/scriptingLog (bool): true LogWidget/statusLog (bool): true LogWidget/warningLog (bool): true MainWindow/geometry (QByteArray): <binary data> MainWindow/mainToolBar.iconSize (int): 18 MainWindow/menuBarGeometry (QByteArray): <binary data> MainWindow/noteSubFolderTreeWidgetExpandState-1 (QStringList): dsv, Programming, Other, Programming/PHP MainWindow/noteTextEdit.code.font (QString): Consolas,10,-1,5,50,0,0,0,0,0,Regular MainWindow/noteTextEdit.font (QString): Consolas,10,-1,5,50,0,0,0,0,0,Regular MainWindow/noteTextView.code.font (QString): Courier New,9,-1,2,50,0,0,0,0,0 MainWindow/noteTextView.font (QString): MS Shell Dlg 2,8.25,-1,5,50,0,0,0,0,0 MainWindow/noteTextView.ignoreCodeFontSize (bool): true MainWindow/noteTextView.rtl (bool): false MainWindow/noteTextView.underline (bool): true MainWindow/noteTextView.useEditorStyles (bool): true MainWindow/noteTextView.useInternalExportStyling (bool): true MainWindow/showNotesFromAllNoteSubFolders (bool): false NoteDialog/geometry (QByteArray): <binary data> NoteDiffDialog/geometry (QByteArray): <binary data> NoteHistory-1 (QVariantList): <variant list with 35 item(s)> NoteHistoryCurrentIndex-1 (int): 34 PiwikClientId (QString): <hidden> SearchEngineId (int): 2 SettingsDialog/geometry (QByteArray): <binary data> SettingsDialog/mainSplitterState (QByteArray): <binary data> SettingsExport/platform (QString): windows ShareDialog/geometry (QByteArray): <binary data> Shortcuts/MainWindow-actionReplace_in_current_note (QKeySequence): Ctrl+H ShowSystemTray (bool): false StartHidden (bool): false TagAddDialog/geometry (QByteArray): <binary data> UpdateDialog/geometry (QByteArray): <binary data> WelcomeDialog/geometry (QByteArray): <binary data> acceptAllExternalModifications (bool): false allowDifferentNoteFileName (bool): false allowNoteEditing (bool): true allowOnlyOneAppInstance (bool): true appMetrics/disableAppHeartbeat (bool): false appMetrics/disableTracking (bool): true appMetrics/notificationShown (QString): true automaticNoteFolderDatabaseClosing (bool): false checkSpelling (bool): false closeTodoListAfterSave (bool): false cryptoKey (QString): <hidden> currentNoteFolderId (QString): 1 currentWorkspace (QString): ae47cf02-04be-42d5-8ffd-f0434f2b0215 cursorWidth (int): 1 customNoteFileExtensionList (QStringList): empty darkMode (bool): false darkModeColors (bool): false darkModeIconTheme (bool): false darkModeTrayIcon (bool): false defaultNoteFileExtension (QString): md demoNotesCreated (QString): true disableAutomaticUpdateDialog (bool): true disableSavedSearchesAutoCompletion (bool): true dockWasInitializedOnce (QString): true enableNoteTree (bool): false enableSocketServer (bool): false externalEditorPath (QString): empty fullyHighlightedBlockquotes (bool): true gitCommitInterval (int): 30 gitExecutablePath (QString): empty gitLogCommand (QString): empty guiFirstRunInit (bool): true ignoreAllExternalModifications (bool): false ignoreAllExternalNoteFolderChanges (bool): false ignoreNoteSubFolders (QString): ^\. imageScaleDown (bool): false imageScaleDownMaximumHeight (int): 1024 imageScaleDownMaximumWidth (int): 1024 initialLayoutIdentifier (QString): minimal insertTimeFormat (QString): empty interfaceFontSize (int): 11 interfaceLanguage (QString): en internalIconTheme (bool): false itemHeight (int): 15 legacyLinking (bool): false localTrash/autoCleanupDays (int): 30 localTrash/autoCleanupEnabled (bool): true localTrash/supportEnabled (bool): true localTrashSplitterSizes (QByteArray): <binary data> markdownHighlightingEnabled (bool): true navigationPanelHideSearch (bool): false networking/ignoreSSLErrors (bool): true networking/proxyType (int): 2 newNoteAskHeadline (bool): false noteEditIsCentralWidget (bool): false noteListPreview (bool): false noteSaveIntervalTime (int): 60 noteSubfoldersPanelDisplayAsFullTree (bool): true noteSubfoldersPanelHideSearch (bool): false noteSubfoldersPanelOrder (int): 0 noteSubfoldersPanelShowFullPath (bool): false noteSubfoldersPanelShowNotesRecursively (bool): true noteSubfoldersPanelShowRootFolderName (bool): true noteSubfoldersPanelSort (int): 0 notesPanelOrder (int): 0 notesPanelSort (int): 1 notesPath (QString): ../../../Dropbox/docs/!notes/qownnotes-notes notifyAllExternalModifications (bool): false overrideInterfaceFontSize (bool): false ownCloud/supportEnabled (bool): false ownCloud/todoCalendarBackend (int): 3 ownCloud/todoCalendarCalDAVPassword (QString): <hidden> ownCloud/todoCalendarCalDAVServerUrl (QString): empty ownCloud/todoCalendarCalDAVUsername (QString): empty ownCloud/todoCalendarCloudConnectionId (int): 1 ownCloud/todoCalendarDisplayNameList (QStringList): empty ownCloud/todoCalendarEnabledList (QStringList): empty ownCloud/todoCalendarEnabledUrlList (QStringList): empty ownCloud/todoCalendarUrlList (QStringList): empty ownCloudInfo/appIsValid (bool): false ownCloudInfo/connectionErrorMessage (QString): empty ownCloudInfo/notesPathExistsText (QString): empty ownCloudInfo/serverVersion (QString): empty previousWorkspace (QString): ae47cf02-04be-42d5-8ffd-f0434f2b0215 recentNoteFolders (QString): D:/!/test restoreCursorPosition (bool): true savedSearches/noteFolder-1 (QStringList): lo, Note 2020-01-15T23.23.48, Note 2020-01-15T23.23.44, Note 2020-01-15T23.23.40, Note 2020-01-15T23.22.47, Note 2020-01-15T23.22.46, Note 2020-01-15T23.22.34, Note 2020-01-15T22.49.53, Note 2020-01-15T22.49.49, Note 2020-01-15T22.49.45, Note 2020-01-15T22.49.31, * Li, Note 2020-01-15T21.54.12 showMatches (bool): true showMenuBar (bool): true showStatusBar (bool): false systemIconTheme (bool): false tagWasAddedToNote (QString): true taggingShowNotesRecursively (bool): false tagsPanelHideSearch (bool): false tagsPanelOrder (int): 0 tagsPanelSort (int): 0 todoCalendarSupport (bool): false toolbar/1/items (QStringList): action_Back_in_note_history, action_Forward_in_note_history, , actionShow_local_trash, actionAllow_note_editing toolbar/1/name (QString): mainToolBar toolbar/1/title (QString): main toolbar toolbar/2/items (QStringList): actionFormat_text_bold, actionFormat_text_italic, actionStrike_out_text, actionInsert_code_block, actionInsert_block_quote toolbar/2/name (QString): formattingToolbar toolbar/2/title (QString): formatting toolbar toolbar/3/items (QStringList): actionInsert_Link_to_note, actionInsert_image, actionInsert_current_time toolbar/3/name (QString): insertingToolbar toolbar/3/title (QString): inserting toolbar toolbar/4/items (QStringList): action_Encrypt_note, actionEdit_encrypted_note, actionDecrypt_note toolbar/4/name (QString): encryptionToolbar toolbar/4/title (QString): encryption toolbar toolbar/5/items (QStringList): actionWorkspaceComboBox, actionStore_as_new_workspace, actionRemove_current_workspace, actionRename_current_workspace, actionSwitch_to_previous_workspace, actionUnlock_panels, , actionToggle_distraction_free_mode, action_Increase_note_text_size, action_Decrease_note_text_size, action_Reset_note_text_size toolbar/5/name (QString): windowToolbar toolbar/5/title (QString): window toolbar toolbar/6/items (QStringList): action_Quit toolbar/6/name (QString): quitToolbar toolbar/6/title (QString): quit toolbar toolbar/size (int): 6 useSoftWrapInNoteEditor (QString): true useUNIXNewline (bool): true webSocketServerService/bookmarksNoteName (QString): Bookmarks webSocketServerService/bookmarksTag (QString): bookmarks webSocketServerService/port (int): 22222 workspace-ae47cf02-04be-42d5-8ffd-f0434f2b0215/name (QString): John Kind workspace-ae47cf02-04be-42d5-8ffd-f0434f2b0215/noteSubFolderDockWidgetVisible (QString): true workspace-ae47cf02-04be-42d5-8ffd-f0434f2b0215/windowState (QByteArray): <binary data> workspace-initial/name (QString): Минимальный workspace-initial/noteSubFolderDockWidgetVisible (QString): true workspace-initial/windowState (QByteArray): <binary data> workspaces (QStringList): initial, ae47cf02-04be-42d5-8ffd-f0434f2b0215

System environment

ALLUSERSPROFILE: C:\ProgramData APPDATA: C:\Users\johnkind\AppData\Roaming CommonProgramFiles: C:\Program Files (x86)\Common Files CommonProgramFiles(x86): C:\Program Files (x86)\Common Files CommonProgramW6432: C:\Program Files\Common Files COMPOSE_CONVERT_WINDOWS_PATHS: true COMPUTERNAME: COMPUTER ComSpec: C:\Windows\system32\cmd.exe DOCKER_CERT_PATH: D:\Docker Toolbox\machine\machines\default DOCKER_HOST: tcp://192.168.99.101:2376 DOCKER_MACHINE_NAME: default DOCKER_TLS_VERIFY: 1 DOCKER_TOOLBOX_INSTALL_PATH: D:\Docker Toolbox DriverData: C:\Windows\System32\Drivers\DriverData HOMEDRIVE: C: HOMEPATH: \Users\johnkind LOCALAPPDATA: C:\Users\johnkind\AppData\Local LOGONSERVER: \\COMPUTER MAGICK_HOME: D:\apps\imagick\modules\coders\ NO_PROXY: 192.168.99.100,192.168.99.101 NUMBER_OF_PROCESSORS: 8 NVM_HOME: C:\Users\johnkind\AppData\Roaming\nvm NVM_SYMLINK: C:\Program Files\nodejs OS: Windows_NT Path: C:\Ruby26-x64\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files\Intel\Intel(R) Management Engine Components\iCLS\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\Git\cmd;C:\Users\johnkind\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\ProgramData\ComposerSetup\bin;C:\Program Files\TortoiseGit\bin;D:\apps\php\;D:\apps\imagick\;D:\Docker Toolbox;C:\Ruby26-x64\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files\Intel\Intel(R) Management Engine Components\iCLS\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\Git\cmd;C:\Users\johnkind\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\ProgramData\ComposerSetup\bin;C:\Program Files\TortoiseGit\bin;C:\Users\johnkind\AppData\Local\Microsoft\WindowsApp;D:\apps\php; PATHEXT: .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.RB;.RBW PROCESSOR_ARCHITECTURE: x86 PROCESSOR_ARCHITEW6432: AMD64 PROCESSOR_IDENTIFIER: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel PROCESSOR_LEVEL: 6 PROCESSOR_REVISION: 9e0a ProgramData: C:\ProgramData ProgramFiles: C:\Program Files (x86) ProgramFiles(x86): C:\Program Files (x86) ProgramW6432: C:\Program Files PSModulePath: C:\Program Files (x86)\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules;C:\Program Files (x86)\AutoIt3\AutoItX PUBLIC: C:\Users\Public SESSIONNAME: Console SystemDrive: C: SystemRoot: C:\Windows TEMP: C:\Users\johnkind\AppData\Local\Temp TMP: C:\Users\johnkind\AppData\Local\Temp USERDOMAIN: COMPUTER USERDOMAIN_ROAMINGPROFILE: COMPUTER USERNAME: johnkind USERPROFILE: C:\Users\johnkind VBOX_MSI_INSTALL_PATH: C:\Program Files\Oracle\VirtualBox\ windir: C:\Windows QML_DISABLE_DISK_CACHE: true

[pbek]

Thank you for reporting, I can reproduce your issue.

[pbek]

Searching is done directly in a in-memory sqlite database.

https://github.com/pbek/QOwnNotes/blob/a7d8e0ed33ee58888ad910b08df5b6e828f4293a/src/entities/note.cpp#L813-L876

Seems like that's a limitation there.

[pbek]

Haven't found a way around it yet.

[pbek]

As far as I found out sqlite only supports case insensitive search with ASCII characters.

[ysaroka]

As far as I found out sqlite only supports case insensitive search with ASCII characters.

Yes it is.

The SQLite API has a sqlite3_create_function(): https://www.sqlite.org/c3ref/create_function.html Probably, you can override or create a new the LOWER (or LIKE) method to search in UTF-8 case-insensitively: ... UTF8_LOWER(note_text) LIKE ? ... or ... note_text UTF8_LIKE ? ...

Found the same problem here: https://forum.lazarus.freepascal.org/index.php?topic=34259.0 This is Pascal, but in this case it does not matter.

[ysaroka]

Another solution is to store additionally "note_text_lowercase" and "file_name_lowercase" fields for the corresponding values in lower case and compare them with the query string reduced to lower case. But i think this is not a good solution)

[pbek]

The SQLite API has a sqlite3_create_function():

I played around with that a bit. The only way to create custom functions in Qt seems to be to depend on (and include) the sqlite source directly. I don't really want to introduce an external dependency to QOwnNotes.

[pbek]

sqlite3.c has 223k lines of code :laughing:

[ysaroka]

I don't really want to introduce an external dependency to QOwnNotes.

I agree, bad idea. Sorry, I thought that Qt SQLite driver has a native ability to define custom functions.

[pbek]

Sorry, I thought that Qt SQLite driver has a native ability to define custom functions.

It's all abstracted behind QSqlDatabase

[macsunmood]

I second that. Are there viable approaches to solving this problem?

[pbek]

Not that I found out any...

[macsunmood]

sqlite only supports case insensitive search

Here's a relatively fresh article "5 ways to implement case-insensitive search in SQLite with full Unicode support": https://dev.to/shallowdepth/5-ways-to-implement-case-insensitive-search-in-sqlite-with-full-unicode-support-53p2

could this help?

also this: https://sudonull.com/post/64260-Another-implementation-of-case-insensitive-search-for-Cyrillic-characters-in-SQLite

[pbek]

We would need to find something that works with Qt5...