Continuous intelligence transforms streaming data into streaming insights. Computing occurs continuously, with insights delivered in real-time driven by data.
The explosive availability of streaming data offers businesses incredible opportunities and challenges. They are increasingly turning to continuous intelligence, which is a form of real-time analytics of streaming data used to support decisions when actions must be taken as changes occur.
To better understand what continuous intelligence is and what it can do for organizations, RTInsights recently sat down with Simon Crosby, CTO at Swim, which offers the first open core, enterprise-grade platform for continuous intelligence at scale. Here is a summary of that discussion.
What is continuous intelligence?
RTInsights: What is continuous intelligence, and how is it different from big data analytics, streaming analytics, and business intelligence?
Crosby: Continuous intelligence lets organizations analyze, then act, and then store later, processing real-time streaming data on-the-fly. Insights are delivered a million times faster by avoiding database roundtrips, and deep insights emerge from the fusion of streaming and traditional data as applications continuously analyze, learn, and predict.
Continuous intelligence lets businesses do many things that deliver significant benefits. Specifically, they can:
Always have the answer: Algorithms are adapted to analyze, estimate, and predict continuously from boundless data streams.
Continuously analyze new data: Each event is processed immediately in the context in which it is relevant, using stateful, in-memory processing for performance.
Analyze and visualize in context: Continuous intelligence discovers meaning in data—real-world relatedness, such as containment, proximity, or correlation—and delivers insights in real-time.
Track data sources over time: Tracking the states of all data sources over time, concurrently, allows applications to discover deep insights.
Find relationships in data: Many data sources are mobile, and their relationships to other sources continually change. Continuous intelligence apps track and analyze changing relationships to find new insights.
A good way to differentiate continuous intelligence is to look at what it does compared to traditional – big data and streaming – analytics methodologies.
While modern databases can store massive amounts of data for later analysis, and update relational tables or modify graphs, continuous intelligence drives analysis from the arrival of data. Businesses can adopt an “analyze-then-store” architecture that automatically builds and continuously executes a distributed, live model from streaming data.
Whereas streaming analytics applications use a top-down UI or query/response user-driven control loop, continuous intelligence applications continuously compute and stream insights, deliver truly real-time user experiences, and facilitate real-time automatic responses at massive scale.
Business Intelligence applications focus on the fusion of many data sources at rest, batch-style, to deliver periodic insights from data. CI applications continuously compute, so they always have the answer.
What is driving the need for CI?
RTInsights: What business factors and developments are driving the need for continuous intelligence?
Crosby: Today, everything has a CPU, and over 20 billion devices are connected per year. But storage, bandwidth, and compute still arent free. Hundreds of millions of users all want personalized experiences online, and IT infrastructure at scale involves thousands of containers and VMs [virtual machines] on-premises and in the cloud. Everything is well instrumented, but the traditional tech built for the “store-then-analyze” era can’t store all data or analyze it quickly enough to deliver insights when they are needed most, which is immediately.
What are the obstacles to using CI?
RTInsights: What are some obstacles to developing continuous intelligence?
Crosby: First, we need to address the challenges of stateless, RESTful microservice-plus-database architecture used in the cloud. Accessing a database to gather state for analysis is a million times slower than a stateful approach that runs in-memory—at CPU speed. So, a stateful architecture optimized for in-memory analysis, learning, and inference is required, given the incessant growth in data volumes.
Second, CI isn’t simply a problem that can be solved with in-memory databases: Numerous open-source projects and commercial solutions offer in-memory databases, data grids, or caching to enhance performance. But none of these approaches addresses the need to compute on-the-fly as data arrives. Databases don’t run applications, can’t understand data, and can’t continuously find relationships between entities.
Third, many have tried (and failed) with event streaming. Technologies such as Apache Kafka help tame access to data streams by letting any number of sources publish events about topics to a broker. Applications subscribe to topics of interest. But the challenge is that like databases, brokers don’t execute applications—they act as a buffer between the real world and applications. Reasoning about the meaning of a set of events requires a stateful system model that captures all entities’ states that the application needs to analyze. In pub/sub speak, this is the role of the stream processor, not of the event streaming infrastructure.
What organizations need is a platform optimized for continuous intelligence. The open-source SwimOS project is an example. Developers use object-oriented Java programs, and the platform builds and scales a computational model from streaming data. Each vertex in the graph it builds is a concurrent, stateful representation of a single source that continuously computes as data arrives.
What’s needed for CI success?
RTInsights: What are the necessary elements for a successful continuous intelligence deployment?
Crosby: CI effectively transforms streaming data into streaming insights. Applications are live computational graphs whose vertices continuously compute as the states of their neighbors change. In this way, computing occurs continuously, with insights delivered in real-time driven by data. The key need is to adopt an architectural approach that permits continuous, stateful analysis, learning, and prediction, which is driven by data—not by a user query or a GUI update.
How is CI being used?
RTInsights: Can you give some examples of how continuous intelligence is being used?
Crosby: CI is used today to monitor network performance, optimize supply chains, improve customer service, find failures in enterprise infrastructure, and enhance security. It lets smart cities continuously analyze, learn, and predict traffic, transit, and utility demands. Finally, it offers mobile service providers a powerful tool to optimize their own operations for dynamic slicing, quality of service, and optimal connection quality.
Learn more about Continuous Intelligence in Swim’s new guide: The Definitive Guide to Continuous Intelligence. Download Now.
