Fast Data = Smart Business: On Real-Time Business Intelligence

There is a long tradition of categorizing business users of BI systems according to their ability to survive in the data jungle. As far back as 1988, in the first published data warehouse architecture, I defined two distinct groupings: (1) dependent users who require full support, assistance, and guidance through all BI facilities, including managers, professionals, and clerical personnel; and (2) independent users, who are experienced in computing systems and include financial analysts, administrators, and business planning specialists.

Since then, business users have become much more computer literate, of course, and many managers and professionals can jockey spreadsheets as well as anybody. However, the data jungle has grown thicker and thornier with new types and sources of data while business pressure for ever faster decisions has grown dramatically. Self-service BI has promised end users the opportunity to design and deploy their own reports and analyses, but distinct roles continue to exist, consumers and power users being common categories. Furthermore, uptake is generally reported to be limited to some 20 to percent of users in most organizations. Today, we are in the early stages of a dramatic change in the BI landscape with the tsunami of fast data coming our way, and that leads to the need for real-time business intelligence.

As data volumes and velocities increased, even as far back as the 1990s, BI practitioners began to talk about strategic, tactical and operational BI, with decision timeframes as shown in Figure 1.

Figure 1: Operational (real-time business intelligence) versus tactical and strategic BI.

In the case of real-time business intelligence (operational BI), traditional business intelligence tools proved less effective and new approaches emerged. Dashboards displayed multiple types of data up to real-time, with predefined causal relationships, and alerting for unusual situations. Decision management systems automated more straightforward decisions, especially on websites—cross-selling, up-selling, small loan offers—using rule-based or decision tree approaches. As a consequence, business users were increasingly distanced from the detailed data. Indeed, the value-add of business users in such operational BI circumstances was considerably reduced. This is also shown in Figure 1, where strategic and tactical BI continue to be divided between consumers and power users, but automation is increasing in operational BI. The timeframe of this figure is 1990 to 2010, although, of course the boundaries were moving toward the right at all levels over this period.

Today, with vast quantities of data arriving at high speed, particularly from the Internet of Things, the roles of business users in decision making must be fundamentally reassessed. Automation of decision making proceeds apace and the rapidly growing role of machine learning and artificial intelligence will further reduce the role of business users in detailed decision making.

While much has been written about technological displacement of jobs, until recently the focus has been on manual and semi-skilled labor. That is already changing. A recent preliminary research report from McKinsey states that up to “45 percent of the activities[of] individuals… can be automated by adapting currently demonstrated technologies… we discovered that even the highest-paid occupations in the economy, such as financial managers, physicians, and senior executives, including CEOs, have a significant amount of activity that can be automated.”

In the past week, Google’s DeepMind division announced that their artificial intelligence program, AlphaGo, has beaten the European champion of Go, a Chinese board game viewed as a much tougher challenge than chess because of the enormous number of possible positions—more than the number of atoms in the universe. University of Cambridge’s Prof Zoubin Ghahramani commented that this breakthrough illustrates: “reinforcement learning – getting computers to learn to improve their behaviour to achieve goals. That could be used for decision-making problems… in businesses or anywhere you’d like to have computers assist humans in decision making”.

Figure 2. Fast data entails a greater role for automated decision making.

The implications for business decision makers are far-reaching. Wherever fast data is prolific, irrespective of industry, automated decision making will assume a far greater role as shown in Figure 2. This will be most evident in real-time business intelligence (operational BI), where we may expect that the vast majority of decision making will be automated with minimal human supervision.

In tactical BI, a significant percentage of decisions could be automated. While strategic BI may continue to exist in its current form for some time to come, we might also expect some automation to occur at this level. We may thus see substantial reductions in the number of decision makers, especially at mid-levels in organizations. Automation certainly leads directly to faster decision making and, through that, to significant business successes for its fastest adopters.

Want more? Check out our most-read content:

White Paper: How to ‘Future-Proof’ a Streaming Analytics Platform
Research from Gartner: Real-Time Analytics with the Internet of Things
E-Book: How to Move to a Fast Data Architecture
The Value of Bringing Analytics to the Edge
Three Types of IoT Analytics: Approaches and Use Cases
IoT Architectures for Edge Analytics
IoT Security: It’s More About Privacy Than Killer Roombas

Liked this article? Share it with your colleagues!