Matrix not PD issue in Variational GP Multiple Inputs/Outputs with Non-stationary kernel.

[Ganesh1009]

I am using Variational GP and using Multidimensional input and output. I created a non-stationary version of RBF kernel k(x,x') * k(x,-x') which works perfectly fine when input is 1D. When the input is multidimensional, I get the below error:

NotPSDError(f"Matrix not positive definite after repeatedly adding jitter up to {jitter_new:.1e}.")
gpytorch.utils.errors.NotPSDError: Matrix not positive definite after repeatedly adding jitter up to 1.0e-06.

after 100 iterations.

Below is my custom kernel:

def postprocess_rbf ( dist_mat ) :
	return dist_mat.div_ ( -2 ).exp_ ( )


class TestKernel ( gpytorch.kernels.RBFKernel ) :

	def forward ( self ,
	              x1 ,
	              x2 ,
	              diag = False ,
	              last_dim_is_batch = False ,
	              **params ) :
		diff = self.covar_dist ( x1 ,
		                         torch.neg ( x2 ) ,
		                         square_dist = True ,
		                         diag = diag ,
		                         postprocess = True ,
		                         dist_postprocess_func = postprocess_rbf ,
		                         **params )
		return diff

where x1 and x2 are 6D rotation representations.

The forward method is as below

	def forward ( self , x ) :

		mean_x = self.mean_module ( x )

		if self.config [ 'kernel' ] == 'non_stationary' :
			covariance_x = self.covariance_module ( x ) * self.covariance_module_ ( x )
		else :
			covariance_x = self.covariance_module ( x )

		return gpytorch.distributions.MultivariateNormal ( mean = mean_x ,
		                                                   covariance_matrix = covariance_x
		                                                   )

and covariance module is defined as below:

self.covariance_module = gpytorch.kernels.ScaleKernel (
					base_kernel = gpytorch.kernels.RBFKernel ( batch_shape = torch.Size ( [ self.config [ 'Dy' ] ] ) ,
					                                           ard_num_dims = self.config [ 'Dy' ] ) ,
					batch_shape = torch.Size ( [ self.config [ 'Dy' ] ] )
					)


self.covariance_module_ = gpytorch.kernels.ScaleKernel (
					base_kernel = TestKernel ( batch_shape = torch.Size ( [ config [ 'Dy' ] ] ) ,
					                           ard_num_dims = config [ 'Dy' ] ) ,
					batch_shape = torch.Size ( [ config [ 'Dy' ] ] )
					)

[wjmaddox]

For one, I don't think you'd need to implement anything custom here:

you could just define the kernel as a function of a standard kernel:

self._covariance_module = RBFKernel()

def covariance_module(self, x, xp=None):
    if xp is not None:
        neg_xp = -xp
    else:
        neg_xp = None
    return self._covariance_module(x, xp) * self._covariance_module(x, neg_xp)

but, more generally, I'm not sure that this kernel would necessarily be positive definite any more and you should probably try to verify if it is (in several dimensions).

[Ganesh1009]

@wjmaddox, if K is stationary, then K'=K(x-y)*K(x+y) -> K' is non-stationary and that is what I have tried to accomplish in my above code. I am not understanding why it would not be a PD anymore.

And, the code does work for a few iterations when the rotation is represented in 6D and after a few iterations, it fails.

The same rotation matrix when represented in quaternion, it works all fine.

Also, I would like to attach a reference issue that we discussed earlier which might make this clear:

https://github.com/cornellius-gp/gpytorch/issues/1843

[wjmaddox]

My issue is that in general, i'm pretty sure K(x - y) isn't positive definite any more. see the counter example https://github.com/cornellius-gp/gpytorch/discussions/1894 .

Can you evaluate the covariance matrix when it fails? If so, how largely negative are the eigenvalues? If they're basically zero, then you can just hackily fix these issues by evaluating the covariance matrix and manually trimming the negative eigenvalues.

[Ganesh1009]

@wjmaddox, thank you for your response. Could you please let me know how can I construct non-stationary kernels?

Regards, Ganesh S. K