Today’s AIOps and observability solutions are applying increasing levels of intelligence to IT Ops data with the goal of generating actionable insights. However, sophisticated AI and ML can sometimes be no match for ‘bad data.’ It’s the age-old problem of garbage in/garbage out. Data can’t simply be taken at face value because it can lead to bad decisions: allocating resources to a digital service that does not need it, chasing the wrong root causes, rolling back a version of software due to phantom problems. You can see the challenge. In a recent survey of IT practitioners by EMA Research, problems with data accuracy were the second leading challenge to AIOps initiatives for the surveyed organizations.
There’s a better way. It begins by cleaning data. IT operations data comes from a variety of sources, often from multiple vendors. Steps must be taken to make the data ready for analysis. Clean data is derived by curating data at the point of collection. Let’s take the example of network observability, which often has variability due to typical multi-vendor network environments. Clean data will be normalized for network vendor differences, then relationship metadata is determined and appended to the payload. Indeed, it’s more important to connect the data than merely collect it – so relationships are key. By cleaning the data it is ready for analysis leading to a reduction in backend processing as well as a reduction in false positives.
Going deeper into the network observability example, SDN networks produce a tsunami of data that is generally disconnected and difficult to comprehend. AIOps and observability solutions are built to handle massive volumes of data, but today’s software-defined environments require hyper-scale. To effectively analyze the volume and variety of data, relationship metadata is needed to connect the SDN overlay with the SDN underlay. Clean data must be curated to derive this relationship at the point of collection and attach the relationship metadata. This approach results in reduced alarm noise, lower operational costs and better detection of network performance anomalies.
Starting with clean data is important, but it is not enough to drive truly actionable insights. Data also has to become smarter. Without smart data, even the most advanced networks still have outages due to a lack of meaningful, symptomatic data.
One way to make IT Ops data smarter is by applying domain expertise. Many domain agnostic AIOps approaches treat data at face value. A better approach is to apply understanding of the particular domain being observed to enrich that data for better insights. The best of both worlds comes from having a domain-agnostic solution that also applies domain-specific intelligence.
Another way to make data smarter is to start with smart data – sometimes bad data can’t be taught new tricks! Today’s hyperscalers and cloud providers need new ways of observing and new data types. These new data types are smart at the start, allowing for tracking of the right metrics and statics to help NetOps and AIOps tools be more proactive in preventing major outages. Collaboration between network equipment providers, service providers and observability vendors can increase the availability of additional, meaningful data types that enable precise understanding and control of network experience.
Smart Data Enables Smart Observability
Observability has emerged as a key capability for organizations that are successfully adopting AIOps. EMA reports that observability (end-to-end visibility) is the most important capability in the AIOps implementation of organizations that are “extremely successful” with AIOps. Observability has shifted from the simple characteristic of an individual element (can you see it) to a broader view of seeing how everything is connected to form a digital service. Said simply, observability is about seeing the forest and not only the trees.
A key characteristic of smart data is that it is connected. In turn, smart observability leverages smart data to tie together the end to end view of a given business service from the end user or IoT device to the backend services and everything in between – even across software-defined and hybrid cloud environments. This enables organizations to move from simply collecting data, checking component status and keeping the lights on towards true observability that understands the connections in the data, the end-to-end digital service health and helps prioritize against business impact.
AIOps and Observability Virtual Summit
Broadcom Software, a leader in AIOps and Observability solutions is sponsoring a free AIOps and Observability Virtual Summit on Sept. 29. Hear from industry experts such as Nick Lippis of ONUG and Andre Kindness of Forrester alongside leaders at organizations such as Broadridge, Telefonica O2 and Fujitsu as they take you on a journey from data to insights.
