typesense-docsearch-scraper
typesense-docsearch-scraper copied to clipboard
feat: configurable batching and rate-limiting for typesense imports
Change Summary
• implement record batching with buffer size and time-based flush triggers
• add buffer_size_limit and flush_interval_seconds config parameters
• support environment variables TYPESENSE_BUFFER_SIZE_LIMIT and TYPESENSE_FLUSH_INTERVAL_SECONDS
• add validation rules ensuring parameters are positive integers
• update typesensehelper constructor to accept batching parameters
• replace immediate record import with buffered approach for better performance
• add example configuration to typesense_docs.json
• update documentation with batching configuration options
performance impact: reduces api calls to typesense by batching records instead of sending them immediately in small chunks
PR Checklist
- [x] I have read and signed the Contributor License Agreement.