Only a small number of companies currently have mature observation processes in place, making it a field ripe for innovation.
Organizations face myriad obstacles when managing observability. Log data has been a critical tool for years and continues to be so. We’re talking petabytes of information if not venturing close to more. Although the explosion in volume has created some new challenges for organizations, it has not dampened enthusiasm for insights.
It’s not only IT stakeholders either. Business users are also leveraging log data to uncover insights about customer activity, the before-mentioned compliance reporting, and improving product and user experiences.
A new research report from Era Software conducted by Dimension Research states, “2022 State of Observability and Log Management,” sheds light on three areas:
- Exploding but critically important log data volumes
- How IT is managing this explosion
- What innovation can do to ease the tension
The report includes responses from 315 IT professionals in various roles from enterprises with at least 100 employees in a variety of industries. Almost half of the respondents pointed to compliance issues as one of the top use cases for analyzing log data, somewhat surprising considering how many specialized reporting tools are now available.
See also: AIOps and Observability Roll into the Next Stage
Observability benefits both IT and business users
Organizations manage log data through either reduction in volume or a cost reduction. The availability of open-source tools to handle storage and other functions seems to be allowing more creative solutions. However, respondents reported trickiness specifically when managing volume itself because it’s difficult to predict what log data will be necessary later. Easy retrieval is another challenge.
Regardless of the method, IT still requires a lot of resources and time to manage log data effectively. One of the biggest frustrations reported in the survey is scalability—even if IT teams can build all the dashboards in the world to monitor log data, scaling to the next phase is currently difficult.
Innovation will be a necessary next step to reducing complexity
The vast majority of respondents cite the need for better tools and processes to relieve stress on IT. Only a small number currently have mature observation processes in place, making it a field ripe for innovation. IT and business users agree that new solutions in the field would benefit both IT and overall business operations.
End-to-end observability and the ability to ingest any form of data are at the top of the wish list. Adopting observability pipelines is still primarily a work in progress but should getting them right would revolutionize the way a company approaches its data.
