stratosphereips
Do a better documentation on how to use the mlflow module, and video examples on how to retrain it in your traffic
Hey @eldraco I read the detection_modules.md file. It has just 1 line about ML flows module. I can add a paragraph about it, similar to other detections. I need more information on how ML flows. We can adda video later once we get traffic that triggers the module.
Looking at the code, I understood following(not everything will go in documentation)
- ML Flow module, is a binary classification.
- It is trained on sets of packets, if found none 2 packets are hardcoded to train
- Training model is saved in bin file, for prediction later
- Training packets, have features like "icmp, icmp-v6, udp, tcp, source ip, destination ip, total packets send, total bytes".
- Packets without ports number are not part of features.
- Module uses SGD classifier for training and testing using partial_fit method.
Yes, please add a new section to that file, but not only one paragraph, because it will not be enough to explain everything. I suggest that you write about it while you use it and get comfortable with it.
- It uses a scaler object to scale all the values to the same range, like a normalization. It is trained during training, and stored in a bin file, so it can be used later in testing. This is important because the scale of the training data should be the same as the scale of the testing data. However, new testing data may arrive outside the scale, so we are careful.
- the model used now is a SGDClassifier() from scikit-learn, which is a linear classifier. This is not a good model, but we use it to test the idea and because scikit-learn can do transfer-learning using this model.
- ICMP and other protocols are not features. Read the code, this are protocols to ignore. The rest are features
- Features must be numeric for now. So no categorical features are used.
- Use use transfer learning to incorporate the packets from users into the model deployed within Slips when you clone the repo. This means that you can 'extend' the original model with your data without loosing the capabilities of the original model.
- Src IP and dst IP are never used, so the model is not biased towards a particular IP.
- The two hardcoded flows must be used, because when you 'transfer-learning' your traffic at home you need some 'benign' and some 'malicious' flow to have a classification model. If you try to train a classification model with only one class, it will fail. So this gives a very small flow as 'starting' point
Also be careful that it was reported that the training was failing now. So I'm fixing it