Call us today on

01782 752 369

Showing posts with label Machine Learning. Show all posts
Showing posts with label Machine Learning. Show all posts

Part 2 Machine Learning


The following Blog Post is written by Brian Steele our Technical Development Lead.

These days, the terms “Machine Learning” and “Artificial Intelligence” are phrases that are bandied around a great deal when vendors talk about the features of their Monitoring Solutions. But “What is Machine Learning?” and “How does Machine Learning help Monitoring Tools?”

In our previous Blog we discussed “What is Machine Learning?”, now we ask “How does Machine Learning help Monitoring Tools?”

Part 2 – How does Machine Learning help Monitoring Tools?

The Goal of Machine Learning in Monitoring

The goal of Machine Learning in monitoring is to reduce the administrative and diagnostic workload of engineers to help them get to the root cause of a problem faster. 

This is achieved by deploying a Monitoring Solution that uses the following main initiatives augmented by Machine Learning to simplify and automate the root cause analysis process:

1. Correlate ALERTS - Identify alerts that correlate to a single root cause alert.
2. Relevant METRICS - Identify which metrics are relevant to diagnosing the root cause issue.
3. WHAT IS NORMAL - learn what is “normal” for any given metric.

If the environment to be monitored is a simple one then a single monitoring tool may be enough. In a more complex environment, multiple tools may be required to collect the different types of metrics needed to perform comprehensive root cause analysis. Either way, a MLMS (Machine Learning Monitoring Solution) can be implemented by deploying a single tool that has ML (Machine Learning) features or by integrating multiple tools into an AI-Ops tool.

Correlate Alerts

We are all used to getting “hundreds” of alerts for the same root cause issue! A Machine Learning Monitoring Solution can help with this problem through the following mechanisms:

1. SERVICE TOPOLOGY – The Machine Learning Monitoring Solution is configured with an understanding of service topology. In other words, it knows the parent/child relationships between configuration items. This understanding of service topology can be configured manually or learned from monitoring tools that have discovered the hierarchy. Because Machine Learning can see the parent/child relationships it will know that if a parent configuration item fails, then there is little point in taking heed of the alerts coming from its child configuration items; so, it mutes them or labels them as being secondary to the root cause alert.



2. ALL ALERTS ALL THE TIME - During model training, Machine Learning looks at all alerts to see if there is any correlation between seeming unrelated alerts. By monitoring all alerts all the time, Machine Learning may observe that two alerts always occur roughly at the same time making them a prime candidate for correlation. This observation can still be true despite the fact that they don’t appear within the same service topology tree and therefore don’t have any parent/child relationship. By correlating alerts in this way, a Machine Learning Monitoring Solution can consolidate many alerts under a single root cause alert reducing the alert cannon. In the following example, the four alerts (A01, A02, A03, and B01) are consolidated to a single root cause alert, which could be A02:




Relevant Metrics
Consider a critical application that is running slow. The application is hosted on a single server so only the metrics from that server are being monitored to determine the health of the application. However; the application is particularly sensitive to the performance of its disc storage which is hosted on a SAN which is in turn serving multiple applications across the wider landscape. Taking all of this into account, it becomes obvious that the availability and performance of the application is dependant on much more than just its hosting server. 

Machine Learning will determine which metrics are relevant to the availability and performance of an application during the selection of the hyper-parameters in the model training phase. In this way, a Machine Learning Monitoring Solution will, without the restriction of human assumptions, have a much more open and comprehensive determination of service availability and performance. This is a massive help when it comes to predicting potential problems (before they manifest themselves) and performing root cause analysis.




What is Normal
For a traditional monitoring tool, the engineer is required to select an appropriate threshold for warning and critical alerts against each metric being monitored. The problem with this approach is that what is an appropriate threshold for one configuration item will be something completely different for another. Therefore, what tends to happen, is that engineers accept the default system-wide thresholds set by the monitoring tool vendor. This gets you by but is inappropriate as a long-term solution because it leads to some conditions not generating alerts while others generate too many alerts. 

A Machine Learning Monitoring Solution will learn the correct threshold for any metric based on its history during normal behaviour. The Machine Learning will also consider the time of day or day of the week and month to cater for regular processes like backups etc... which are expected to have an impact. This is a great help in keeping the Monitoring Soluton sufficiently sensitive to issues while reducing or eliminating false alerts.




In Conclusion

  • You don’t have to configure what data to monitor; Machine Learning will work that out for you.
  • You don’t have to configure rules and thresholds; Machine Learning will work that out for you.
  • Machine Learning will predict results even if the metrics have never occurred before
  • The predictions improve over time as the Machine Learning observes more edge cases.

If you have found this introduction interesting then download our mini-presentation here to get more insights into Machine Learning including the fundamentals of Neural Networks.

Part 1 What is Machine Learning?


The following Blog Post is written by Brian Steele our Technical Development Lead.

These days, the terms “Machine Learning” and “Artificial Intelligence” are phrases that are bandied around a great deal when vendors talk about the features of the monitoring solutions. But “What is Machine Learning?” and “How does Machine Learning help Monitoring Tools?”

This is a two-part Blog that aims to answer these questions starting with “What is Machine Learning?”:

The Goal of Machine Learning


The goal of Machine Learning is to take some existing real-world data (like monitoring metrics for CPU, Memory, and Disc, etc…) along with a target metric (like device availability, application performance or user experience) then work out the following:

  • Which of the real-world data is important? So, for example… By examining the CPU, Memory and Disc utilisation of a server over a two-week period, I may be able to determine that only the CPU utilisation has any correlation to the observed server application performance. So, the Machine Learning has to work out that it needs to focus on the CPU utilisation, and can afford to ignore all other metrics if it wants to be able to predict the availability and performance of the server in question.
  • What is the relationship between the real-world data and the target metric? So, for example… By examining the CPU, Memory and Disc utilisation of a server over a two-week period, I should be able to develop a formula (function) that takes these metrics as parameters and then provides an answer which closely matches the observed availability and performance of the server in question.
  • Accept new, unseen, real work data and predict what the target metric is likely to be So, by monitoring the required metrics from (1) and feeding them into the formula developed in (2) I should be able to take any new unseen set of metrics for a given server and then calculate the expected availability and performance.

What does Machine Learning look like?


The graph below helps us to visualise what is going on inside Machine Learning in a scenario where it has worked out that it only needs to monitor the CPU percentage utilisation (Horizontal axis) to predict the server latency (Vertical axis). Remember, it has done this by observing real-life data, nobody had to sit down and program this understanding through the configuration of rules.



The Machine Learning has further observed that it is able to group servers together by their role which enables a more accurate prediction of expected latency for any given server as a function of its CPU utilisation and server role. The Machine Learning also develops a formula (function) which will draw a line through the data points to closely approximate the observed results. This line can then be used to predict the expected results from any intermediate point where we have previously had no data to guide us.

We don’t want to memorise results we want to calculate them


In the following example, we see that the “Web Servers” have results that don’t follow a straight line through the other server roles. In this case, the Machine Learning will develop a more complex formula (function) which will closely follow the observed results while still allowing us to predict the expected results for any intermediate point. During the development of the formula (function), the Machine Learning continually compares its predicted results with the real-world results to see how closely it is tracking. Once the formula (function) is within a close enough approximation it is considered to be “Right Fitting” and the development of the formula (function) stops.



By contrast… in the following example, the Machine Learning has overdeveloped the formula (function) so that the predicted results exactly match the real-life data. This is called “Over Fitting” and means that the Machine Learning has basically “Memorised” the results. This means that its effectiveness will be compromised when it comes to predicting the results for intermediate points.


In Conclusion

  • You don’t have to configure what data to monitor; Machine Learning will work that out.
  • You don’t have to configure rules and thresholds; Machine Learning will work that out.
  • Machine Learning will predict results even if the metrics have never occurred before.
  • The predictions improve over time as the Machine Learning observes more edge cases.

If you have found this introduction interesting then download our mini-presentation here to get more insights into Machine Learning including the fundamentals of Neural Networks.

Part-2 in this series will explore the topic “How Does Machine Learning Help Monitoring Tools?”

Popular Posts

Blog Archive

Our Clients

  • Camden
  • Vodafone
  • SCC
  • timico
  • William Hill