Operational intelligence, which is the real time next phase of Business Intelligence, enables companies to improve operations in real time.
Real-time analytics is having a profound business impact on companies across many domains. I asked Venkat Venkataramani, Co-Founder and CEO of Rockset to share some thoughts on the matter.
Bhat: Batch-based data warehouses have been getting the job done well for years. What is motivating companies to move from batch to real-time analytics?
Venkataramani: Real-time interaction is becoming our baseline expectation in our consumer and business lives.
Remember when you shopped online and it took a week or more for your packages to arrive, and you were fine with it? I now expect to get my shipment confirmations within minutes of my credit card being charged and to be able to track my two-day delivery as soon as it leaves the warehouse. I expect the same with my Grubhub dinner delivery and my Uber pickup.
This shows that fast analytics on fresh data is better than slow analytics on stale data. Fresh beats are stale every time. Fast beats slow in every space.
Time and time again, companies in a wide variety of industries are boosting revenue, increasing productivity, and cutting costs by making the leap from batch analytics to real-time analytics.
Also, you can’t transform into a digital enterprise and build a data-driven culture relying on batch-based analytics and BI.
There is too much latency at every step — finding the data, ingesting it, querying it, and representing it. In an age of lightning-fast consumer apps such as Instagram, users won’t tolerate excruciatingly-slow analytics experiences. Not your customers, nor even your internal employees. If answering every question takes 20 minutes, then your workers simply won’t ask any follow-up questions.
Where extract-based BI tools fail, modern interactive analytics tools and data-driven customer-facing applications succeed, providing users with sub-second response times as they drill down into seconds-old data.
Users embrace a data-driven culture when they can ask questions in real time. Being able to explore data for answers, also known as guided decision-making, is incredibly powerful. It enables companies to pull off bold and creative moves which, because they are informed by the latest data, don’t come with the normally-associated risks. Smart data-driven decisions become a company-wide habit. And that can only happen with real-time analytics.
See also: Continuous App Availability Requires Predictive Insights
Bhat: Doesn’t the so-called cloud-native modern data stack, which succeeded the big data analytics stack, take advantage of streaming data and other real-time data sources?
Venkataramani: The modern data stack does ingest real-time data. But, many of its key components, including the centerpiece data warehouse, are still fundamentally batch-based. That’s true even if it is a cloud data warehouse like Snowflake or BigQuery. There’s too much latency and delay in data pipelines as data travels from ingestion to querying.
Bhat: What are some concrete use cases of companies using real-time analytics to deliver more business value and make or save money?
Venkataramani: There are several, including:
- Logistics: As soon as you drop off a package for shipping, a sensor in the smart dropbox feeds the data to the shipping company, which detects which driver is closest and reroutes them for immediate pickup. Every day, millions of job tickets are created and tracked in real time across air, freight rail, maritime transport, and truck transport.
- Fitness leaderboards: 10,000 steps a day is a fine goal, but most of us need more motivation. One fitness company I know has an app that gives users coins for steps. It also updates leaderboards in real time for a little friendly competition.
- Fraud detection: Time is of the essence in cybercrime. To minimize risk, real-time data such as credit card transactions and login patterns must be constantly analyzed to detect anomalies and take swift action.
- Customer personalization: Online shoppers like relevant product recommendations, but they love it when they are offered discounts and bundles for them. To deliver this, e-tailers are mining customers’ past purchases, product views, and a plethora of real-time signals to create targeted offers that customers are more likely to purchase.
- To compete effectively today, e-commerce companies must go beyond simply price, selection, and convenience. Personalizing their customer experience is a must-have. Statistics show that 80 percent of shoppers are more likely to buy from brands that offer personalized experiences. Personalization can also increase sales by 20 percent. And with the wealth of customer data and real-time signals available today, most e-tailers are rushing to take advantage.
All of these use cases require not just real-time data but an entire set of tools to ingest, prepare, analyze and output it instantly. Enter the modern real-time data stack, a new wave of cloud solutions created specifically to support real-time analytics with high concurrency, performance, and reliability — all without breaking the bank.
Bhat: Is there any use case that you consider to be the most promising use of real-time data?
Venkataramani: Operational intelligence, which is the real-time next phase of Business Intelligence.
Analyzing last quarter’s sales on a static dashboard is classic BI – retrospective and passive.
By contrast, operational intelligence provides rich insights from both historic and up-to-the-second data, enabling you to affect and improve any part of your operations in real time. And whereas BI leverages a data warehouse and associated ETL/ELT tools, operational intelligence is produced by a scalable real-time analytics stack that delivers business observability.
Business observability systems provide a real-time view into the health of mission-critical operations within the enterprise. Early business observability systems focused on making real-time monitoring smarter by analyzing multiple datasets at a time. This helped to spot potentially-catastrophic anomalies early while minimizing false positives that create alert fatigue in workers.
State-of-the-art business observability delivers even more. It can deliver instant answers to complex queries in many areas: cybersecurity, logistics, sales, and any other domain that is key to your company’s bottom line. Those insights also help workers uncover and fix the root causes of problems faster than ever and make decisions that maximize productivity, increase revenue and minimize downtime.
Bhat: What are some examples of organizations deploying business observability to glean operational intelligence?
Venkataramani: There is a fast-growing Buy Now, Pay Later (BNPL) company with 100+ million customers and hundreds of thousands of merchants worldwide. Their online payments service, being its sole revenue source, is mission-critical. Monitoring for problematic payments and failed transactions is key. If all the Apple Pay payments for a large retailer in Switzerland are suddenly down, that can wipe out hundreds of thousands in lost sales.
The company had a cloud data warehouse that crunched transaction data through anomaly detection models every six hours. As the company grew and the number of transactions increased, these anomaly detection jobs started exceeding six hours to complete. The company wanted to upgrade to a real-time monitoring system that could comb through millions of payments per day and trigger instant alerts without a flood of false positives.
The company built a business observability system around a real-time analytics database. The database continuously ingests transaction data and provides up to the second accurate real-time metrics across a wide range of dimensions, including merchant, payment method, region, time of day, etc. The data is analyzed against statistical models every minute instead of every six hours. The company’s incident response team is notified immediately of serious issues. That team is equipped with analysis of both recent and historical payments data so it can investigate and resolve the issue quickly.
Operational intelligence is saving this BNPL company tens of millions of revenue losses from downtime every year. Even more importantly, it allows them to immediately alert their customer, in this case, that large online retailer, and protect their business too, thereby increasing customer loyalty and decreasing churn.
Another example is the American company Command Alkon, which provides a SaaS logistics service for construction companies and their suppliers. Millions of concrete shipping deliveries a day are tracked by Command Alkon in real time, giving its customers granular visibility into the status of their shipments, as well as instant alerts if shipments may be delayed. This is real-time operational intelligence that Command Alkon shares with customers.
There is also Seesaw, a popular K-12 e-learning platform, that built a real-time business observability system that provides operational intelligence to a variety of roles. They include engineers monitoring the platform’s health, salespeople researching usage by school district employees to ensure satisfaction of Seesaw’s paying customers, and executives looking for visually-informed trend data to plan Seesaw’s product roadmap. Seesaw’s system also provides key insights to external users such as teachers and school principals who need to track in real-time how many students are turning in assignments.
All three of the above companies built their business observability systems around a real-time analytics database, choosing this over a cloud data warehouse, NoSQL database, or a pre-built business observability solution.
Bhat: Why not deploy a packaged business observability solution to glean operational intelligence instead of building it?
Venkataramani: The operational intelligence stack is still in its infancy. Most observability solutions still focus on specific domains and teams within an enterprise. Even with that narrow view, they are expensive, inflexible, and non-interoperable with other enterprise systems.
Their high cost, immature features, and inflexibility outweigh any turnkey promises. After carefully weighing the costs, risks, and benefits, many companies are choosing to build their own operational intelligence systems using a real-time analytics database as the core.
What’s the most surprising example of an industry or field that is taking advantage of real-time analytics to glean operational intelligence?
Venkataramani: Everyone’s heard of Uber, Meta, and Airbnb.
But how about the NFL? During the Super Bowl, I marveled at the coaches using sophisticated data analytics for real-time in-game strategizing.
Those ubiquitous Microsoft Surface tablets are much more than digital playbooks. Players and coaches could quickly review video clips to analyze mistakes in just-completed plays and diagram better ones. More powerfully, every play was categorized, tagged, and crunched by an analytical engine to highlight trends. Coaches could pull that real-time data themselves. Or, as two-thirds of NFL coaches are doing, they could call on their team’s analytics expert connected via headset to provide data-driven guidance on crucial decisions like fourth downs, two-point conversions, etc.
The NFL is miles ahead of the Fortune 500 when it comes to real-time data-driven decision-making? They are no longer watching game films on Mondays.
Monday Morning Quarterbacking no longer suffices in pro football. The same goes for stale business intelligence. As corporate operations increasingly rely on technology and data, so does their financial health. Businesses can no longer afford operational bottlenecks, much less any downtime. Slow is the new down. Alerts – when they result in long outages – are too late. That’s why companies need operational intelligence.
