Anomaly Detection

Problem statementTo build a user and entity behaviour analytics (UEBA) system using Machine Learning models which can detect anomalies based on login time, day, history of access type of login and account type etc. in user login behaviour.  The model must automate the prediction and receive feedback from the analyst on the final label, to be then used for training. The system should also integrate seamlessly with the existing tech stack of the client to pull data and push predictions.

Solution: We developed a system that predicts anomalies using machine learning models consuming active directory logs. It integrates as a microservice with the ELK stack used at client setup to collect and aggregate log data. The system then identifies and categorizes incidents and events, as well as analyses them. The software delivers on three main objectives, which are to

  • Provide reports on security-related incidents and events, such as successful and failed logins, malware activity, and other possibly malicious activities.
  • Send alerts if an anomaly is detected by the system.
  • Provide provision to give feedback & push predictions for the anomalies.

Techniques & Algorithms: 

Machine learning 

  • Heavy training algorithms – deep neural networks
  • Real-time prediction algorithms
  • Statistical calculations

Backend & Data Engineering.  

  • Model Training Infra
  • Model Prediction API
  • API Security

Key Takeaways : 

  • Extremely fast training – <3 seconds per user for all models
  • Our algorithms achieved 99.9% percent F-score.
  • The False Positive rate was .1 % ie. only 1 out of 1000 predictions were not anomalies.
  • Sub second dwell-time.  Prediction time was. <200ms between anomaly event and prediction
  • The system eliminated manual task of security analyst by detecting the anomalous and rare activity of a user in real time and sends alerts to the security analyst
  • The system provided a user and entity behavior analytics dashboard to monitor user behavior.

We are a team of machine learning programmers, data engineers and strategy analysts.We have no legacy… and that’s our biggest strength we do what we are passionate about, question the status quo and are fresh and unconventional. Our team is skilled at data sciences, algorithms and consulting. We have delivered radical solutions which have been accomplished only because of the unique blend that we bring as a group.

Scientist Technologies ®


Ground Floor, Indiqube Penta, #51 Richmond Road, Ashok Nagar, Bengaluru, Karnataka 560025

+91-991 669 51 56

Copyright © 2018