Localdocs changes

[manyoso]

This adds the non-ui portion of localdocs changes necessary for the major new ui redesign branch. This includes several commits that when taken together upgrade the localdocs database.

It includes:

• Backing out changes that are in main for mixpanel events of localdocs/database

• Adding a fixme reminder that will need to be resolved before this hits main

• Removing the debug example for database

• The meat:

  Large update to localdocs with many moving parts. This commit is a breaking change that requires re-indexing of all collections.

  • Adds a general mechanism for handling breaking changes to localdocs database upgrades

  • Upgrades the database format to version 2 and includes numerous changes to the underlying tables including: a) adds word and token count fields for chunk table b) adds a boolean field indicating whether the chunk has a generated embedding c) adds a last updated time field for collection table which can be displayed in GUI d) adds an embedding model field for collection table e) adds a boolean field for collection table to allow forced re-indexing

  • Adds several statistics about the collection which can be displayed in GUI a) total documents b) total words c) total tokens d) last updated time e) which document is currently being indexed f) which document is currently being embedded g) which model is being used to generate embeddings for the collection

  • Simplifies communication between the database thread and the GUI thread removing a lot of extra code and makes clear what methods/signals in database class are soley used for communicating state to the GUI

  • Provides a pool for sending embedding requests to the embedding thread to

  • Provides a pool for sending embedding requests to the embedding thread to increase speed of embedding when the chunks per document are small

  • Provides a mechanism for resuming indexing/embedding when the app shutsdown or crashes in the middle of indexing/embedding

  NOTE: This change is part of a larger set of commits, but can be built and compiles up to this point

This change is requested to be merged into the newly created major_new_ui_redesign branch which is a feature branch and not main. When ready, the major_new_ui_redesign branch will be reviewed and merged into main.