[Feature Request]: Implement Caching in loaded datasets for Improved Performance

[J-B-Mugundh]

trafficstars

Is there an existing issue for this?

• [X] I have searched the existing issues

Feature Description

• To enhance the performance of our Streamlit app, especially when handling large datasets, we can implement data caching. This change allows us to:
• Reduce Loading Times: By caching data loading functions, we minimize the time required to fetch data during user interactions, leading to a smoother user experience.
• Improve Efficiency: Functions that are resource-intensive, such as data preprocessing and loading from external sources, will now return cached results, reducing redundant computations and improving overall efficiency.

Lazy Loading: The app will now lazily load datasets, allowing only the necessary data to be fetched when required. This further optimizes memory usage and speeds up the initial loading time.

Use Case

The app provides users with various functionalities, including visualizations of salary trends, comparisons between different departments, and filtering options based on experience levels and job titles. The dataset can be substantial, with thousands of records spanning several years.

By implementing caching for functions that load and process the salary data, you can significantly enhance the app's performance.

For instance:

• Data Loading: When a user interacts with the app for the first time, the data is loaded from a CSV file. With caching, subsequent interactions that require the same data do not trigger the loading process again; instead, the cached data is utilized.
• Data Processing: If the app includes complex calculations (e.g., calculating mean salaries based on user-selected filters), caching these results ensures that the calculations are only performed once per session unless the underlying data changes.

Benefits

• Reduced Latency: Cached data retrieval eliminates the need for repeated loading, leading to immediate response times when users interact with the app.
• Improved User Experience: Users can explore data visualizations and insights without waiting for extensive loading times, making the app more engaging.
• Lower Resource Consumption: By avoiding unnecessary data processing and loading, the app consumes fewer server resources, making it more efficient.
• Scalability: The app can handle larger datasets without degrading performance, making it suitable for various applications.

Priority

Medium

Record

• [X] I agree to follow this project's Code of Conduct
• [X] I'm a GSSOC contributor
• [X] I want to work on this issue
• [X] I'm willing to provide further clarification or assistance if needed.