Buffered writes

[tharropoulos]

TLDR

Add buffer-based operations for Typesense to improve reliability and performance.

Change Summary

• This change adds a buffering layer to improve reliability of Typesense operations
• Buffer stores operations in Firestore before processing them in batches
• Provides retry capabilities for failed requests and better error handling
• Configurable via extension parameters for flexibility

Added Features:

1. Buffer functionality for Typesense operations:

   • Added option to use buffering for Typesense operations with a new config parameter: TYPESENSE_USE_BUFFER
   • Buffer stores operations in Firestore before processing them in batches
   • Implemented automatic retry mechanism for failed operations

2. New Configuration Parameters in extension.yaml:

   • TYPESENSE_USE_BUFFER: Toggle buffering on/off
   • TYPESENSE_BUFFER_COLLECTION_IN_FIRESTORE: Collection name for buffer storage
   • TYPESENSE_BUFFER_BATCH_SIZE: Number of documents to process in a batch
   • TYPESENSE_BUFFER_MAX_RETRIES: Maximum retry attempts for failed operations
   • TYPESENSE_BUFFER_FLUSH_INTERVAL: Cron-style interval for buffer processing

3. New files:

   • processBuffer.js: Scheduled function to process buffered operations
   • Added test files: indexOnWriteWithBuffer.spec.js and indexOnWriteSubcollectionWithBuffer.spec.js

Code Changes:

1. In indexOnWrite.js:

   • Modified to check for buffer setting and route operations accordingly
   • Split functionality into realTimeWrites and bufferedWrites
   • Added handling for path parameters in buffered operations

2. In processBuffer.js:

   • Implemented scheduled function to process buffer in batches
   • Added retry logic for failed operations
   • Added status tracking (pending → processing → completed/retrying → failed)

3. In config.js:

   • Added new configuration parameters with defaults

4. Test Environment Updates:

   • Added support for custom Typesense fields in test environment
   • Created comprehensive test suites for buffered operations
   • Added test environment configurations for buffer testing

5. Fix: Double URL Encoding:

   • Removed redundant encodeURIComponent calls when accessing Typesense collections after updating typesense-js
   • Updated all test files to match this change

Dependencies:

1. Updated Typesense client:
   • Bumped from v1.8.2 to v2.1.0-3 for improved functionality

PR Checklist

• [x] I have read and signed the Contributor License Agreement.