Solution: We developed a system that predicts anomalies using machine learning models consuming active directory logs. It integrates as a microservice with the ELK stack used at client setup to collect and aggregate log data. The system then identifies and categorizes incidents and events, as well as analyses them. The software delivers on three main objectives, which are to
- Provide reports on security-related incidents and events, such as successful and failed logins, malware activity, and other possibly malicious activities.
- Send alerts if an anomaly is detected by the system.
- Provide provision to give feedback & push predictions for the anomalies.
Techniques & Algorithms:
- Heavy training algorithms – deep neural networks
- Real-time prediction algorithms
- Statistical calculations
Backend & Data Engineering.
- Model Training Infra
- Model Prediction API
- API Security
Key Takeaways :
- Extremely fast training – <3 seconds per user for all models
- Our algorithms achieved 99.9% percent F-score.
- The False Positive rate was .1 % ie. only 1 out of 1000 predictions were not anomalies.
- Sub second dwell-time. Prediction time was. <200ms between anomaly event and prediction
- The system eliminated manual task of security analyst by detecting the anomalous and rare activity of a user in real time and sends alerts to the security analyst
- The system provided a user and entity behavior analytics dashboard to monitor user behavior.