typesense-docsearch-scraper icon indicating copy to clipboard operation
typesense-docsearch-scraper copied to clipboard

feat: configurable batching and rate-limiting for typesense imports

Open tharropoulos opened this issue 6 months ago • 0 comments

Change Summary

• implement record batching with buffer size and time-based flush triggers • add buffer_size_limit and flush_interval_seconds config parameters • support environment variables TYPESENSE_BUFFER_SIZE_LIMIT and TYPESENSE_FLUSH_INTERVAL_SECONDS • add validation rules ensuring parameters are positive integers • update typesensehelper constructor to accept batching parameters • replace immediate record import with buffered approach for better performance • add example configuration to typesense_docs.json • update documentation with batching configuration options

performance impact: reduces api calls to typesense by batching records instead of sending them immediately in small chunks

PR Checklist

tharropoulos avatar Sep 02 '25 16:09 tharropoulos