A Business Intelligence Strategy for Real-Time Analytics

Games such as chess and Go display perfect information, and AI clearly has an upper hand. What happens, however, when information is imperfect and requires strategy?

It has often been said that traditional business intelligence is like driving while looking in the rearview mirror. The implication, of course, is that real-time analytics is more like sensible driving behavior, where you keep your eyes mainly on the road ahead.

Both driving an automobile and running a business are all about understanding the environment—both internal and external, and making decisions. That includes setting your direction to reacting in a second to get to where you want to go. Let’s look how the autonomous vehicle metaphor works when it comes to business intelligence strategy, then extend it to other business sectors.

Connected Cars and Artificial Intelligence

Three key technology trends are driving the rapid emergence of partially and, eventually, fully autonomous vehicles: improved, cheaper sensors, expansive Big Data, and major advances in artificial intelligence. All three trends are clearly also in play for the digitalized business. In the case of the auto, the first successful DARPA Grand Challenge for autonomous vehicles took place just over 10 years ago, while Google’s famous project revved up as recently as 2008. Major vehicle manufacturers have, of course, also taken up the challenge, and former hacker, George Hotz built a self-driving car in his garage last year and demoed it to Ashlee Vance of Bloomberg Business. The car’s artificial intelligence (AI) software “learned” to drive by “watching” Hotz for 10 hours on the highway and reportedly drove as well as a teenager learning to drive. Hotz believes that he can eventually offer the combined hardware and software package for as little as $1,000.

Ready to start your journey to becoming an AI-first enterprise?

This impressive progress depends on extensive, real-time data availability and its automated, immediate use in decision making and action taking. With a growing store of contextual information (such as maps, rules, etc.) and a human to set the destination, presumably by voice input, the result is a viable and useful system that clearly undertakes autonomous “reasoning” and decision making, leading to actions that require no human intervention. Irrespective of some more recent and poorly formed arguments about legal responsibility and ethical choices being made by algorithms, the economic drivers are clear to the software giants and reeling automobile manufacturers. How long it takes for autonomous vehicles to become pervasive is immaterial; the direction and end result are clear. And the changes in the economic and social fabric of Western society will be fundamental and far-reaching.

From Self-Driving Cars to Digital Business

Just as with autonomous vehicles, replacing the majority of human decision making with algorithms both removes the need for humans in daily decisions and concentrates the definition of “good” vs. “bad” decisions in the hands of a small coterie of AI experts and designers. On the IT side, consequences reach deep into every aspect of designing, building, implementing and maintaining the entire computing infrastructure of the enterprise, because the majority of existing systems were designed in an era of much lower levels of automation and real-time action. Let’s explore a little further.

In modern business, data flows in from a wide array of physical sensors and online user interactions in large volumes and at high speed. As a direct result of such speeds and volumes, the analysis of this data, as well as the data-driven decisions and actions, must be almost fully automated, with no more than rare exceptions being brought to human attention. The automobile example demonstrates clearly the ability of combined sensor and AI systems to recognize real-world situations and act accordingly.

However, analysis of the text, audio and video from human-sourced information is also improving rapidly. IBM Watson, for example, recently announced the availability of Emotion Analysis that can detect joy, fear, sadness, disgust and anger in text input. All these systems are self-learning, so that the exceptions that require human intervention become rarer as the algorithms improve their performance through deep learning techniques based on observing how humans handle exceptions. Google’s recent impressive success with AlphaGo against Go world champion, Lee Se-dol, is built squarely on such deep learning. The role of the business user in such operational BI/analysis, therefore, is to set direction and handle (increasingly rarer) exceptions.

BI Analysts as “Exception Handlers”

Returning to the auto analogy to look at the data architecture, the structure is unlike a traditional layered computing environment and is closer to that seen in real-time analytics environments. This is hardly surprising, given that the autonomous vehicle system is a real-time algorithm-driven environment. Thus, training data can be extracted for in-depth analysis and modeling off-line, but real-time decision making occurs on the basis of the live data.

In the business world, the data environment is considerably more complex. Real-time data is only a part of the story, although large in terms of data volume and increasing in importance. However, the legally binding transactional data (I call it process-mediated data in my book Business unIntelligence) that forms the basis of the day-to-day operations and management of the business has very different characteristics, requiring integration and reconciliation before use in tracking performance, identifying problems, and setting directions. Use of such historical and, to a lesser extent, current data led to the rearview mirror analogy mentioned earlier. Its importance may diminish with more predictive data available, but it is unlikely we will rip out all the rearview mirrors.

Continuing with the auto analogy, what will be the role of those who used to be drivers in the future? Beyond being merely passengers, most observers expect them to be the exception handlers as discussed above, at least in the transition phase from a human to a hybrid and then fully autonomous driving regime.

There are economic issues with this approach, especially in cases where drivers are an expense, such as in haulage and transportation. Travis Kalanick, CEO and founder of Uber, would allegedly “replace human Uber drivers with a fleet of self-driving cars in a second,” clearly not considering the need for drivers as exception handlers. Google’s own self-driving car initially had no controls, although it was later forced to add them, at least for testing. The main design challenge here, however, is the handover of control in exceptional circumstances from the autonomous vehicle to the human, who is rather unlikely to be concentrating on the situation at the time. The same concern arises in the case of operational BI/analytics systems: how to apprise the overseer of the details of a particular problem fast enough and with all needed information without overwhelming him/her.

Where Algorithms Fit With Tactical, Strategic BI

Of course, not all decision making in business is, or needs to be, real-time. Traditional BI focused first on tactical decision making support—dealing with out-of-the-ordinary situations in slower moving metrics that can typically be analyzed and addressed in time periods from days to weeks. As we move from operational analytics/BI through tactical and onto strategic BI, we move essentially from a world of near-perfect information to one of imperfect information. Games such as chess and Go display perfect information—the state and rules of play are known—and AI is increasingly superior to humans in this field. Driving an automobile and operational decision making certainly do not exhibit perfect information. However, with extensive sensor data availability and relatively short time frames for decision making, we see—only in autonomous vehicles, so far—the ability of AI to perform well.

With tactical and strategic decisions, we move toward what’s called a partially observed world. Nowhere near full information is available, and real-time data has less importance. Rules of engagement and possible outcomes are unclear and susceptible to change. Human motivation plays a significant role. Combined, all these factors imply that AI will be slower to encroach upon these types of BI.

However, Demis Hassabis, vice president of engineering leading Google’s general AI projects, says: “Ultimately we want to apply this [the techniques used in AlphaGo] to big real-world problems…” including initially healthcare, smartphone assistants, and robotics.

Business decision making is not mentioned, although it is certainly a big, real-world problem. Nonetheless, some proportion of tactical decisions are straightforward enough to be already amenable to augmentation (support via algorithmic methods) and we can be certain that proportion will increase. In some special cases, automation may even be possible, especially in simpler and regular tactical BI needs. In fact, Michael Gonzales goes as far as to state in Cognitive Computing: Transforming the Analytic Landscape: “it can be argued that artificial intelligence literally means business intelligence”

The future strategy for BI in tactical and strategic decision making therefore includes extensive use of algorithms. In the earlier stages, this will be less obvious than in the case of operational BI. However, the impact may be just as great when decision makers, especially at the executive level, augment their intuition (or lack thereof) with powerful algorithms that offer novel insights and unexpected correlations. Consider, for example, the advantages accruing to one executive in a negotiation when an emotion recognition algorithm informs her that her counterpart is lying. It will be in subtle, but powerful, ways like this that future BI strategy will begin to appear in tactical and strategic decision making support.

Five Key Considerations for Future BI Strategy

Based on the above thinking, we can propose the following five key considerations to be incorporated into future BI strategies:

In operational intelligence and real-time decision making, business users will increasingly focus on direction setting for automated and semi-automated algorithmic processes that handle large volumes of high-speed data input, rather than being involved in monitoring dashboards
When automated decision making systems detect exceptions they cannot handle and invoke human decision making, the handover must occur early enough to allow human engagement, provide sufficient and suitable background information, and offer insightful suggestions on how to handle the situation
With the ever-tighter integration of decision making and action taking in real-time decision making situations, hybrid operational/informational systems will become the norm, meaning that stand-alone BI systems will largely disappear in the operational BI/analytics domain
Traditional BI approaches to tactical and, especially, strategic decision making will remain important and will not be fully displaced by automated systems, but these will need to take human factors from emotions to motivation more deeply into account.
Artificial intelligence and cognitive computing approaches will increasingly augment tactical and strategic decision making with insight and correlations that are non-obvious to human perception, with BI systems moving from simply displaying data in visual form to taking on a more “conversational” role with human decision makers