evidently
evidently copied to clipboard
evidentlyai
Slow test execution and metric calculation
When using real data with a size of 100k rows and a large number of columns, metrics, and tests (around 1000), their calculation can take up to 20 minutes. Additionally, computer resources are not fully utilized, with a powerful processor not exceeding 20% of one core's capacity. Consequently, with many tests and metrics and a high RPS of new data, Evidently may not be able to process them in time.
I checked the insides of ClassificationPreset and DataDriftPreset. I've seen a lot of data copying, which is far from ideal. Also pandas is used here, while in some cases faster alternatives could be utilized. But imo the most inefficient part is embedding actual data into html reports
I wonder if metric calculation can be done at least in several processes
@nick-konovalchuk
The best solution, I believe, would be to use parallel execution, but we need to explore the feasibility of its application. Trying to optimize individual sections is of little use because in my case, we are calculating ~2000 different tests and metrics.
I don't see any problems with generating HTML since you're only using HTML when necessary.
Also using polars with lazy calculations instead of pandas can be a good solution if we are talking about calc optimization
@c0t0ber Personally I've never used polars, but I think I remember it using all the cores of a CPU. In such setting multiprocessing would be harmful.
@c0t0ber I don't see a problem with generating HTML. I wish they also had an option of generating actual plotly objects, that I can display using streamlit, for instance. I still can display HTML in streamlit. The problem is embedding redundant data in HTML. Do you really need all data point to draw a histogram given that you can't change the bin size after the report is generated? Because as far as I understand they embed ALL data points for ClassificationProbDistribution and charts.
Also I'm less sure about it, but the data points in HTML may be duplicated in context of several metrics/tests
Idk if this is correct and/or possible, but the following would be cool
- Generate actual Plotly objects
- Extract HTML from them when the report is rendered. I think this HTML won't have redundant data embedded