Compound Alerts Within Database Observability Drive Actionable Alerting
Orinal source post by Dominique Flannery
The latest edition to the SolarWinds® Observability Platform, Database Observability, is live!
Previously, you’ve relied on Database Manager to help monitor your database and generate alerts, and while it was ahead of its time for years, the time has come for the next generation of database monitoring.
Key features of Database Observability are:
- Single platform to view the entire network and database as opposed to stitching together numerous tools and services
- The ability to isolate where your data lives geographically
- Compound alerts – can create alerts based on if and statements to improve the quality of alerts, decrease the number of “false positive” alerts, and increase actionable alerts
The compound alerting feature is what I want to focus on in this post, though. In my opinion, compound altering is a game changer. Let me share a real-life example of when compound alerting could’ve helped me identify and solve an issue quicker.
In a previous role as a senior DBA, we often encountered a situation where we’d have too many connections on the database. Basically, database sleep connections were high, database running or active connections were below a certain threshold, and the web application’s CPU was high. When this happened, everyone panicked about too many connections to the database. I quickly noticed the issue was sleep connections. However, nobody believed sleep connections were the issue.
In this scenario, the database itself was not the issue. I noticed the issue was sleep connections but couldn’t always isolate where the problems originated. To figure it out – and prove it was sleep connections, I had to spend weeks gathering data, building charts, and going through other teams’ systems to build a report to illustrate the issue.
Sleep connection means there is a connection to the database, but the connection isn’t doing anything. In response, the web application’s CPU is high, and it is taking longer to go through the code to close those connections out. As a result, they sit there idle, not doing anything. When this happens, the connections eat up database resources like memory and available connections and can cause errors.
Holding the connection time longer may not seem like a huge deal, but each connection is allocated with a certain amount of memory. Every database has a threshold of total connections – if this threshold is exceeded, recovery can take much longer, and if the connection is slow, load times are slower. End users can end up frustrated or faced with errors and other issues.
With Database Observability, instead of triggering an alert based on a single event or threshold, you can create compound alerts designed to notify based on a combination of conditions. This can reduce false positives and ensure the alerts you receive are more actionable. To do so, you would create alerts with defined individual conditions using logical operators (AND, OR, NOT) to combine them into a compound condition. As a result, instead of alerting when just one metric is off (like CPU usage is high), a compound alert might trigger only when CPU usage is high AND memory usage is above 90% AND disk I/O is slow.
If I had been using Database Observability, I could have created a compound alert such as, “Database sleep connections are high AND database running or active connections are below a certain threshold AND the web application’s CPU is high.” With this alert, I could pinpoint the issues much quicker than I was able to do before. I could’ve also generated reports within Database Observability to illustrate the issue to leadership.
Altogether, compound alerts enable you to create parameters to get information about situations right off the bat. Customers are thrilled to utilize this extremely helpful and valuable feature to create actionable and informational alerting.
The post Compound Alerts Within Database Observability Drive Actionable Alerting appeared first on Orange Matter.