Small data observability can provide the real-time insights that matter, empowering companies to maximize the uptime of complex and growing infrastructure.
Capturing data for systems observability is a growing priority for organizations. Not only is it important for maximizing IT efficiency, but it is also key to identifying and responding to issues with security and performance. When it comes to SRE, IT operations, and other critical teams, capturing and analyzing the right data to draw conclusions and make informed decisions makes a huge difference in their speed and effectiveness. As more data is generated at the edge, it is pushing teams to reevaluate their observability strategy.
The most common approach companies take involves moving raw edge data to a central repository and analyzing it in batches – a “big data” observability strategy. However, this has become infeasible – or at the very least, very expensive – at scale. The world produced an estimated 79 zettabytes of data in 2021, leaving many companies overwhelmed by the expenses of storage and analysis in the form of software-as-a-service (SaaS) contracts and cloud expenses.
The nature of the edge, and data on the edge, also presents unique concerns. Companies face the challenge of configuring, deploying, and maintaining agents in thousands of locations at once in order to extract all the data being produced. Said locations might simply be altogether incompatible with edge visibility solutions that depend on on-premises hardware or VMs for observability. There’s simply no room for these solutions on an IoT device or lean branch location server.
There is an alternative approach to enabling observability on the edge – push analysis out to where the events happen and pair it with the ability to dynamically control what is analyzed in real time. We call this “dynamic edge observability,” or what you may call a “small data” strategy. This approach empowers teams to conduct and retrieve analyzed data in real time, scaling commensurately with the raw data edge infrastructure generates. This offers unrivaled specificity and flexibility, speeding up incident response while keeping costs stable even as companies analyze more data.
Still, big and small data aren’t mutually exclusive: big data can provide the context needed to better leverage the hyper-specific small data approach. Let’s take a closer look at how big and small data approaches to observability compare, how small data can dynamically generate insights, and how the two approaches can work in tandem.
See also: Legacy Systems Hindering Unified Observability Rollout
Explaining Big and Small Data Observability Strategies
The big data approach to observability involves bringing raw telemetry from the edge to a central repository where a SaaS or cloud provider can perform analysis on it to generate insights. When faced with the question of how much data should be collected, teams often default to thinking “as much as possible” in the hopes of having what they need when the time comes to ask questions.
Unfortunately, this strategy has significant drawbacks. Transiting and storing so much data is expensive – as is paying for that data to be analyzed – and often, only a fraction of the data winds up offering meaningful insights. The process is often time-consuming, and waiting for analysis introduces a lag between identifying problems and having the insight needed to resolve them. Many firms generate so much data that they run out of quota or storage space, forcing them to make do with short retention times or pay more to ingest and store more data, driving unreasonable costs.
In contrast, a dynamic edge observability strategy analyzes the raw data streams right where the data is generated and outputs only “small data” – smartly aggregated metrics and logs that provide direct insight. Those results are available locally for immediate action and are also collected centrally for global visibility.
Small data also addresses some of the cost and logistical concerns of big data. Since each edge agent emits a very small amount of data as compared to a central repository of raw data, companies are less likely to run out of quota. Processing smaller amounts of data is faster and less costly, especially at scale. The focus on collecting more signal and less noise means that less money is spent to store inactionable data.
Making the Case for Small Data
Edge agents can slice data along any dimension you need to investigate, enabling teams to analyze a subset of relevant data instantly. As a result, small data can allow for rapid response across a range of use cases where big data would be too slow and expensive.
As an example, imagine a retail chain with hundreds of stores has a single malfunctioning cash register at one location. Analyzing that malfunction with a big data approach may work, but as you won’t know which one out of thousands of machines needs to be analyzed and fixed, you would need to be collecting all data from all machines at all times. Instead, with a dynamic edge observability strategy, the operations group responsible for the repair could have the edge observability agent collect deep analytics about the specific cash register’s IP address only when the problem occurs. From there, the group could rapidly diagnose misconfigurations or other issues that are taking place, then remove the request for the extra analysis, returning to baseline observability.
Small data is also useful for responding to potential malicious activity. If a DNS provider notices a huge spike in traffic coming from a particular country, it can set edge agents to provide more details about the top queries coming from that country. From there, reviewing top requests could indicate bot traffic, suggesting a DDoS attack, or otherwise merit further investigation.
Staying Above Water in a Flood of Data
More data at the edge means more potential insights, but it can also result in more resources spent on analyzing data. And in the end, the insights produced may not be timely or valuable. Even if an organization has found a way to handle its current data volume, it will be a huge challenge to scale a big data approach going forward without also integrating a small data observability strategy. Small data observability can provide the real-time insights that matter, empowering companies to maximize the uptime of complex and growing infrastructure.
