marella
Memory optimization
Out of curiosity, I tried containerizing the package with the offline models, installing the required PIP packages and a sample document (size: 62 kB) loaded through chatdocs add.
The build could not complete locally due to memory issues. I tried it in a hosted environment and I got the following warnings during the build process:
Even though the package does not have an official support for containers yet, this indicates that we need to optimize the code perfoming various operations (like adding files or loading models) so that it works well on low end PCs.
The default models in chatdocs.yml require at least 8 GB RAM to run. I picked the smallest models possible that have decent quality. You can use models smaller than the default ones by adding them to your chatdocs.yml file but the response quality might not be good. So it is a trade-off between quality and performance.
