Ikeda Map Learning

[AminMoradiXL]

First off, great work! Second, I'm trying to learn the "Ikeda" map using KAN. My question is do you have any suggestions on improving this? Also this runs too slow (with about 50k data, 2 input, 2 output), and cuda is not working. Do you have any recommendation? Also does lr matter? Here is the code:

So for Ikeda mapping I generated a dataframe using:

import numpy as np
def Ikeda(T, dt, N_sims, a=0.4, u=0.9, x0=0.1, y0=0.1):
    N_t = int(T // dt)
    sims = np.zeros((N_sims, N_t, 2))
    sims[:, 0] = np.array([x0, y0]).T
    for i in range(1, N_t):
        theta = 0.4 - (6 / (1 + sims[:, i-1, 0]**2 + sims[:, i-1, 1]**2))
        cos_theta = np.cos(theta)
        sin_theta = np.sin(theta)
        sims[:, i] = np.array([1 + u * (sims[:, i-1, 0] * cos_theta - sims[:, i-1, 1] * sin_theta),
                               u * (sims[:, i-1, 0] * sin_theta + sims[:, i-1, 1] * cos_theta)]).T
    return sims.astype(np.float32)

import matplotlib.pyplot as plt

def scatter_plot_ikeda_trajectories(trajectories):
    plt.figure(figsize=(8, 6))
    for traj in trajectories:
        plt.scatter(traj[:, 0], traj[:, 1], s=1)
    plt.title("Ikeda Map Trajectories (Scatter Plot)")
    plt.xlabel("X")
    plt.ylabel("Y")
    plt.show()

T = 10000 
dt = 0.01 

N_sims = 1 

ikeda_trajectories = Ikeda(T, dt, N_sims)

scatter_plot_ikeda_trajectories(ikeda_trajectories)

ikeda_trajectories = ikeda_trajectories[0]

import pandas as pd 

df = pd.DataFrame(ikeda_trajectories[:-1,:], columns=['x_n', 'y_n'])

df['x_n+1'] = ikeda_trajectories[1:,0]
df['y_n+1'] = ikeda_trajectories[1:,1]

df.to_pickle('ikeda_1M.pkl')

Then I use the following code to test 1-step prediction and multi-step prediction using KAN:

from kan import *
import torch
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import root_mean_squared_error as rmse
import warnings
warnings.filterwarnings("ignore", message="std()*")
from aux_funcs import *

image_folder = 'video_img'
aux_funcs(image_folder = image_folder)

df = pd.read_pickle('ikeda_1M.pkl')

perc = 3
df = df.iloc[-int(len(df)*perc/100)-1:,:]

df_seen = df.iloc[:int(0.95*len(df)),:]
df_unseen = df.iloc[int(0.95*len(df)):,:]

X = df_seen[['x_n','y_n']].to_numpy()

y = df_seen[['x_n+1','y_n+1']].to_numpy()

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state= 42)

dataset = {
    "train_input": torch.tensor(X_train),
    "train_label": torch.tensor(y_train),
    "test_input": torch.tensor(X_test), 
    "test_label": torch.tensor(y_test)
}

model = KAN(width=[2,10,2], grid=40, k=3, seed=0)

model(dataset['train_input'])
model.plot(beta=100)

results = model.train(dataset, opt="LBFGS", steps=50, lamb=0.01, lamb_entropy=10., save_fig=True, beta=100, lr = 0.01,
            in_vars=['x_n', 'y_n'],
            out_vars=['x_n+1', 'y_n+1'],
            img_folder=image_folder);

y_test_arr = y_test
y_pred_arr = model(dataset['test_input']).detach().numpy()

error_seen = rmse(y_test_arr, y_pred_arr) 
print(f'1-step Prediction error is = {error_seen}')

aux_funcs.plot(y_test_arr, y_pred_arr, t = 100)


###
### Multi-step
###

X_unseen = df_unseen[['x_n','y_n']].to_numpy()

y_unseen = df_unseen[['x_n+1', 'y_n+1']].to_numpy()

T = 1000

y_pred_unseen = np.zeros((T,2))

y_pred_unseen[0,:] = model(torch.tensor(X_unseen[0,:].reshape(1, -1))).detach().numpy()

for i in range(T-1):
    pred = model(torch.tensor(y_pred_unseen[i,:].reshape(1, -1))).detach().numpy()
    y_pred_unseen[i+1, :] = pred
    
y_unseen_trun = y_unseen[:T,:]


error_unseen = rmse(y_pred_unseen, y_unseen_trun)
print(f'Multi-step Prediction error is = {error_unseen} (UNSEEN data)')

aux_funcs.plot(y_unseen_trun, y_pred_unseen, t = 100)

For one step prediction these are the results: 1-step Prediction error is = 0.2624390218891965

For multistep prediction these are the results: Multi-step Prediction error is = 0.6604029929865323 (UNSEEN data)

[KindXiaoming]

Hi, I don't have much experience for KAN + time series, but it seems to me that in

model = KAN(width=[2,10,2], grid=40, k=3, seed=0)

grid is too large. (I'm not sure though since your plot shows some fractal behavior, which requires large grid) And

model.train(dataset, opt="LBFGS", steps=50, lamb=0.01, lamb_entropy=10., save_fig=True, beta=100, lr = 0.01,
            in_vars=['x_n', 'y_n'],
            out_vars=['x_n+1', 'y_n+1'],
            img_folder=image_folder);

try reduce lamb if you don't care much about sparsity/interpretability.

For efficiency, you may reduce sample from 50k to 5k. You may also pull the latest repo which should allow you use cuda.

Also, here's some advice on how to tune hyperparameters. Hope this helps.

[yuedajiong]

Just based on my limited knowledge, for time-series data, it composed by: trend + periods+ noise periods, is proper to be fitted by periodic function/as basis function, especial for high-freq sin(BigCoef*x)， the backgroud is Fourier series.
Otherwise, if you want to fit just based on squeeze-like activation, any form of functions compoise/nn network, you need wider parameters or deep layers.

Of couse, the sin/cos internal is still taylor_expansion, coef befor var x; to computer, no shortcut, just approximation. based on limited +*/ opeators.

I have submited some discussions about special functions, not only periodic function. (as issue, but not real issue, just discussion.)

[AminMoradiXL]

@KindXiaoming Thank you very much for your response. I will implement your suggestions and update on results.

@yuedajiong Good perspective. I will take a look at your issues/discussion too. Thanks!