filter2 command is too slow

[y9c]

Compared with filter command or awk, fiter2 command is much slower, especially for rule with multiple conditions.

It might be relative to this function in the for-loop, which repeatedly parsing the expression. https://github.com/shenwei356/csvtk/blob/9407f73e2d72dddf5042c7dbb6299a180ea9cf4a/csvtk/cmd/filter2.go#L370-L376

[shenwei356]

Yes, I noticed that. It is slow :(

[y9c]

Can we move the Expression parsing function outside the for-loop and run it only once?

[shenwei356]

It is slow, but it must be done like that. Cause filterStr1 is different in each iteration.

[y9c]

Why filterStr1 is different? Can we cache the parsed results?

[shenwei356]

It's the expression, like '$age > 18', the $age needs to be replaced with the value of each row.

[y9c]

Yes. I mean can we parsed the expression as something like '$1>18' and reuse the code of the filter command to deal with the computation afterward

[shenwei356]

parsed the expression as something like '$1>18' and reuse the code of the filter command I don't think so.

God, it's really slow~ I used it a lot recently. Have to improve it, when I have time ~