feat: add batch processing and partitioning to IndexedDB operations

[mksglu]

Data Partitioning in IndexedDB Operations

This pull request introduces a significant feature to our library: Data Partitioning in IndexedDB operations. This enhancement is aimed at improving performance and efficiency when handling large datasets. Below are the key aspects of this feature:

• Efficient Data Handling: Implements batch processing for large datasets. By loading and processing data in manageable chunks (partitions), the library now handles large volumes of data more efficiently, reducing the memory footprint and enhancing performance.

• Scalability: This feature is crucial for scalability. It allows the library to maintain high performance and efficiency, even as the dataset grows. This is particularly beneficial for applications dealing with large-scale vector search operations.

• Configurable Batch Size: The feature introduces a configurable batch size for data processing, enabling fine-tuning according to specific memory and performance requirements of the application.

• Enhanced Data Retrieval: With partitioning, data retrieval is now more targeted. It allows for fetching only relevant subsets of data, which is especially useful in scenarios where data is categorized or segmented into different groups or types.

• Optimized Performance: Overall, this feature contributes to a significant reduction in the memory usage and an increase in the processing speed, thereby optimizing the performance of database operations within the library.

To Do

Implement Automatic Batch Size Adjustment for IndexedDB Operations

As part of our ongoing efforts to enhance the library's efficiency and performance, a future enhancement is planned to introduce automatic batch size adjustment in IndexedDB operations. This feature will further optimize data processing and resource management when working with large datasets. Key aspects of this planned feature include:

• Dynamic Batch Sizing: The feature will dynamically adjust the batch size based on real-time performance and memory usage metrics. This allows for more intelligent and efficient data processing, particularly under varying load conditions.

• Memory Usage Monitoring: Incorporating real-time memory usage monitoring to determine the optimal batch size. This ensures that the library remains within safe memory limits while maximizing data processing throughput.

• Performance Optimization: Automatically optimizing the batch size to balance the trade-off between memory usage and processing speed. This helps in achieving an optimal performance for different sizes and types of datasets.

• Enhanced Scalability: With automatic batch size adjustment, the library will be better equipped to scale with varying dataset sizes, making it more robust and versatile for different use cases.

• User Customization: Although the batch size will be adjusted automatically, user customization options will be provided for advanced use cases where manual tuning is required.

This feature is expected to significantly improve the way our library handles large-scale data operations by making it more adaptive and efficient.