Are you still using techniques like 5 Whys, fish-bone diagram or even guessing for root cause analysis? Would you like to have a root cause instantly with high accuracy? If yes, then read further!
Detecting anomalies, and finding the root cause of the anomaly, is an important application in the field of machine learning in nearly all sectors and disciplines. An anomaly may mean different things in different applications, for example, fraudulent use of credit cards or suspicious transactions in the financial sector, a specific disease or the outbreak of a disease in healthcare, the signs of intrusion in a computer network, a fault in a production system or product in the manufacturing industry, an error in a business process, etc.
Machine learning discovers patterns in the data and therefore it is well suited for discovering unusual patterns which are the signatures of anomalies.
Anomaly detection can be combined with root cause analysis (RCA) in a machine learning (ML) model, or ML anomaly detection may assist traditional RCA techniques in finding the root cause. Machine learning automated RCA may also be useful for the following reasons:
- To perform complex RCA several domain experts are often needed, who may not always be available or may be expensive or difficult to deploy. In this case ML automated RCA can also be considered as a knowledge management tool.
- ML automated RCA may save a lot of time because traditional RCA projects may often take even several days.
There are three main types of machine learning models (methods) which can be used for anomaly detection and/or RCA:
- Supervised,
- Unsupervised and
- Semi-supervised learning based method.
For supervised learning we need labeled (classified) data. In the case of anomaly detection, the data must contain data records labeled as ‘normal’ and data records labeled as ‘anomaly’. We can combine anomaly detection with root cause analysis by defining several types of anomaly labels which all identify a specific root cause. For example, in the case of the root cause analysis of a defective product in a production process, we could define root causes as “wrong material”, “handling error”, “machine error”, etc. instead of just using “wrong product”.
Supervised machine learning based anomaly detection and root cause analysis is a very powerful technique, but collecting and labeling the data is a lot of work and must be done by domain experts. You may also consider using a semi-supervised method (see later) in order to decrease the amount of work needed for labeling the data.
One of the most important parts of anomaly detection and root cause analysis is the data collection and feature selection phase. It is very important to select the right features which can be used to distinguish between normal and anomalous phenomena. In most cases (especially if root cause analysis is integrated) domain experts must design the data collection and feature selection process, and they also must take care of the labeling of the data records if supervised learning models are used.
Usually a traditional structured and systematic approach is used to investigate anomalies and their root causes (when appropriate) and in order to determine the best set of features used in the input data for the machine learning model.
It is often possible to simulate a real situation or environment and introduce all forms of potential anomalies and record the attributes (responses) of the system. It is much easier and faster to study the signatures of anomalies in this way than to monitor real systems. Try to think about a solution for modeling your real situation in some way and try to simulate potential anomalies.
This article is a slightly modified excerpt from the book “The Application of Artificial Intelligence”. If you are interested in the subject then it is strongly recommended to read the book which contains many more details and real world case studies for several sectors and disciplines! The book explains several examples step-by-step by using the AI-TOOLKIT. The book is going through the publishing process at the time of writing this article. You may use the contact form for information about pre-ordering the book.
Learn about the application of Artificial Intelligence and Machine Learning from the book "The Application of Artificial Intelligence | Step-by-Step Guide from Beginner to Expert", Springer 2020 (~400 pages) (ISBN 978-3-030-60031-0). Unique, understandable view of machine learning using many practical examples. Introduces AI-TOOLKIT, freely available software that allows the reader to test and study the examples in the book. No programming or scripting skills needed! Suitable for self-study by professionals, also useful as a supplementary resource for advanced undergraduate and graduate courses on AI. More information can be found at the Springer website: Springer book: The Application of Artificial Intelligence.
|
 |
