Network Anomaly detection on datasets NSL-KDD, Kyoto University and Mawii labs

This project has been conducted under the supervision of Dr. Jinoh Kim and Dr. Donghwoon Kwon at Texas A&M University-Commerce. The research outcome will be published in the proceeding of IEEE ICNC 2018, with the title of “An Empirical Evaluation of Deep Learning for Network Anomaly Detection”.

• Below results are for NSL-KDD Dataset only. Master branch contains code for NSL-KDD dataset. There are separate dev branches for Kyoto University and Mawii labs. The networks implemented are same for all datasets.

Exploratory Data Analysis

Andrew Curves (High dimensional data plots)

T-SNE (Data dimensionality Reduction)

Pattern evolving during epochs

Pattern in final (4000) epoch

Results of Train/Test cycles

Fully Connected Neural Network

Variational Autoencoder

latent variables used for prediction

Variational Autoencoder

Anomaly labels treated as part of actual data

Network learns to regenerated labels treating it as missing data during testing.

LSTM Seq2Seq

Softmax layer is used to convert output sequence to Normal/Anomaly prediction.

Conclusion