libsvm
libsvm copied to clipboard
NU_SVC training is affected by order of training samples
I have originally discovered this effect using C version of libsvm, wrapped into C++ code, using real data.
Then I created a minimal snippet to reproduce it using libsvm in sklearn with random data for training and testing.
So far I have checked only NU_SVC and C_SVC, and it seems that C_SVC is invariant to order of training samples, but NU_SVC isn't, which is counterintuitive (at least for me, since there's no notion of sample ordering in SVM problem formulation).
Here's a snippet to reproduce:
import numpy as np
from sklearn import svm
# training/testing set params
n = 1000 # num of samples
d = 2 # num of dimensions
# random training set (balanced)
train_feats = np.random.uniform(size=(n, d))
train_labels = np.array([i < n/2 for i in xrange(n)]).reshape(n, 1)
# combine to retain connection between features and labels during shuffling
train = np.hstack([train_feats, train_labels])
# 2 svms to compare
csvm = svm.SVC(kernel='rbf',
gamma=0.5,
degree=3,
cache_size=100,
tol=0.001,
C=1.0,
shrinking=False)
nsvm = svm.NuSVC(kernel='rbf',
gamma=0.5,
degree=3,
cache_size=100,
tol=0.001,
nu=0.1,
shrinking=False)
# random testing set (balanced)
test_feats = np.random.uniform(size=(n, d))
test_labels = [i < n/2 for i in xrange(n)]
for i in xrange(10):
# shuffle train set
train_shuffled = train.copy()
np.random.shuffle(train_shuffled)
# split into features and labels
shuffled_train_feats = train_shuffled[:, :-1]
shuffled_train_labels = train_shuffled[:, -1]
# train both svms individually
csvm.fit(shuffled_train_feats, shuffled_train_labels)
nsvm.fit(shuffled_train_feats, shuffled_train_labels)
# make predictions individually
pred_c = csvm.predict(test_feats)
pred_n = nsvm.predict(test_feats)
acc_c = sum(pred_c == test_labels)/float(n)
acc_n = sum(pred_n == test_labels)/float(n)
print acc_c, acc_n
And corresponding output:
0.502 0.49
0.502 0.495
0.502 0.504
0.502 0.5
0.502 0.526
0.502 0.496
0.502 0.505
0.502 0.507
0.502 0.533
0.502 0.516
First column - accuracies using C_SVC, second - NU_SVC using different shufflings of training samples. As you can see C_SVC is invariant to sample order, NU_SVC is not.
I would rather use NU_SVC, since it is simpler to perform grid search on, but this effect makes it almost pointless. Am I missing something?
I think due to numerical inaccuracy, both c-svc and nu-svc may be slightly affected by the order of data. However, the resulting model (or test accuracy) should be similar
Alexey Rozhkov writes:
I have originally discovered this effect using C version of libsvm, wrapped into C++ code, using real data.
Then I created a minimal snippet to reproduce it using libsvm in sklearn with random data for training and testing.
So far I have checked only NU_SVC and C_SVC, and it seems that C_SVC is invariant to order of training samples, but NU_SVC isn't, which is counterintuitive (at least for me, since there's no notion of sample ordering in SVM problem formulation).
Here's a snippet to reproduce:
import numpy as np from sklearn import svm
training/testing set params
n = 1000 # num of samples d = 2 # num of dimensions
random training set (balanced)
train_feats = np.random.uniform(size=(n, d)) train_labels = np.array([i < n/2 for i in xrange(n)]).reshape(n, 1)
combine to retain connection between features and labels during shuffling
train = np.hstack([train_feats, train_labels])
2 svms to compare
csvm = svm.SVC(kernel='rbf', gamma=0.5, degree=3, cache_size=100, tol=0.001, C=1.0, shrinking=False)
nsvm = svm.NuSVC(kernel='rbf', gamma=0.5, degree=3, cache_size=100, tol=0.001, nu=0.1, shrinking=False)
random testing set (balanced)
test_feats = np.random.uniform(size=(n, d)) test_labels = [i < n/2 for i in xrange(n)]
for i in xrange(10): # shuffle train set train_shuffled = train.copy() np.random.shuffle(train_shuffled)
# split into features and labels shuffled_train_feats = train_shuffled[:, :-1] shuffled_train_labels = train_shuffled[:, -1] # train both svms individually csvm.fit(shuffled_train_feats, shuffled_train_labels) nsvm.fit(shuffled_train_feats, shuffled_train_labels) # make predictions individually pred_c = csvm.predict(test_feats) pred_n = nsvm.predict(test_feats) acc_c = sum(pred_c == test_labels)/float(n) acc_n = sum(pred_n == test_labels)/float(n) print acc_c, acc_n
And corresponding output:
0.502 0.49 0.502 0.495 0.502 0.504 0.502 0.5 0.502 0.526 0.502 0.496 0.502 0.505 0.502 0.507 0.502 0.533 0.502 0.516
First column - accuracies using C_SVC, second - N_SVC using different shufflings of training samples. As you can see C_SVC is invariant to sample order, NU_SVC is not.
I would rather use NU_SVC, since it is simpler to perform grid search on, but this effect makes it almost pointless. Am I missing something?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.*
In this case it looks like a bug - accuracy variation is too high in the NU_SVC case
csvc and nusvc are equivalent. Are you using the corresponding parameters? Alexey Rozhkov writes:
In this case it looks like a bug - accuracy variation is too high in the NU_SVC case
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*