activitysim icon indicating copy to clipboard operation
activitysim copied to clipboard

pandas 1.3 PerformanceWarning

Open jpn-- opened this issue 4 years ago • 0 comments
trafficstars

Running with pandas version 1.3 emits PerformanceWarnings at various points, they look like this:

WARNING - activitysim/activitysim/abm/models/util/mode.py:123: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider using pd.concat instead. To get a de-fragmented frame, use newframe = frame.copy() choosers[trace_column_names] = choosers.index

As noted in #449 the defrag approach suggested in the warning does not work on DataFrames with a lot of bool columns. We should identify whether this is actually a meaningful performance problem for us, and if it is then investigate possible solutions.

jpn-- avatar Jul 09 '21 14:07 jpn--