pykan icon indicating copy to clipboard operation
pykan copied to clipboard

Updated auto_symbolic to include a configurable threshold & simplicity weight

Open srigas opened this issue 6 months ago • 0 comments

Dear Ziming, even though KAN 1.0 was an impressive work on its own, I believe that you managed to up the ante even further with KAN 2.0. Impeccable work, I will keep following the updates and working with KANs! This PR is a proposition:

  • Motivation: After trying to train relatively small KANs to fit relatively simple expressions, I noticed that auto_symbolic would sometimes produce seemingly nonsensical results. For example, when plotting the KAN, a certain edge would clearly correspond to a sine function, however auto_symbolic would set it to zero. Upon further investigation, I realized that auto_symbolic calls suggest_symbolic with the default value for the weight_simple parameter, which prioritizes low complexity over high R^2 values.

  • Proposition: I added the weight_simple as a parameter of the auto_symbolic function as well, so that a user can choose how to prioritize symbolic fixing across the entire architecture. Using suggest_symbolic and fix_symbolic manually for each edge is tedious, especially in large architectures. Additionally, I introduced a r2_threshold parameter which can control which edges to replace with symbolic expressions based on the achieved R^2. If the threshold is not reached, the user is prompted to either keep training until a better fit is found, or simply lower the threshold. Setting r2_threshold = 0.0 reproduces the original implementation, so there is complete "backward compatibility".

Kind regards, Spyros Rigas.

srigas avatar Aug 21 '24 18:08 srigas