Can this be used for anomaly detection?

[sambalshikhar]

[TimyadNyda]

Hi,

Yes, of course. The purpose of autoencoders is to compress the data, then rebuild it. Along this process, the main properties of the data are kept (in order to reduce the loss on average), however unexpected variations, that are more than noise, and happen according to random patterns (if this is an acceptable definition of an anomaly), are removed in the reconstruction phase.

Really, think of it as a process summarizing the main idea of your data.

So, if your network if good enought to rebuild the main shapes of your data, while removing these unexpected variations, then you could expect it to remove anomalies (by focusing on where the reconstruction error is 'big', i.e > a certain threshold carefully chosen).

[sambalshikhar]

I have tried to replicate this https://arxiv.org/pdf/1910.03818.pdf paper using your code,however I could only get a F1 score of about 0.45 .I tried using various threshold.I tested it on Yahoo S5 dataset .It would be great if you could mention what are the changes I could make to implement this paper

On Mon, Mar 16, 2020 at 3:54 PM Dany Yatim [email protected] wrote:

Hi,

Yes, of course. The purpose of autoencoders is to compress the data, then rebuild it. Along this process, the main properties of the data are kept (in order to reduce the loss on average), however unexpected variations, that are more than noise, and happen according to random patterns (if this is an acceptable definition of an anomaly), are removed in the reconstruction phase.

Really, think of it as a process summarizing the main idea of your data.

So, if your network if good enought to rebuild the main shapes of your data, while removing these unexpected variations, then you could expect it to remove anomalies (by focusing on where the reconstruction error is 'big', i.e > a certain threshold carefully chosen).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Danyleb/Variational-Lstm-Autoencoder/issues/5#issuecomment-599456807, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJF4BUHFSGSBQW3724IQA2LRHX45TANCNFSM4LKPAZUA .