By processing data immediately as it arrives, businesses can better understand what customers want and need right now instead of yesterday, last month, or last year.
Businesses today recognize the importance of making strategic, data-based decisions, yet many struggle to make this a reality. In fact, 69% of C-level technology and business executives report that they have yet to create a data-driven organization, according to a 2019 NewVantage survey. One challenge is that the data management strategy of many businesses is to collect as much data as possible in a data lake, data warehouse, or another repository, then figure out how to make sense of that data afterward.While a tolerable approach for legacy applications, reporting and analytics which were limited to – and thus designed for – “batch” processing, the opportunities for capitalizing on data have changed drastically in recent years.
The problem with focusing on accumulating as much data as possible is that it doesn’t necessarily result in a more complete view of operations and customers. In fact, the complexity of dealing with an excess of data can cloud visibility. And trying to tease out ever-more-subtle insights from ever-growing amounts of historical big data can become a tremendous challenge in its own right. Instead, it’s become important for today’s data-driven solutions and applications to take immediate action on data as it streams into the enterprise. The new critical challenge is successfully taking advantage of this world of fast data.
What is fast data?
Fast data introduces the idea that by processing data immediately as it arrives, businesses can better understand what customers want and need right now instead of yesterday, last month, or last year. Instead of making static plans based on past information, they can dynamically align strategic vision with current information to better serve customers, increase revenue, and improve efficiency. In contrast to legacy batch-driven approaches, fast data focuses on immediacy to transform, refine, and analyze data as it arrives, enabling users, applications, and services to act on that information quickly.
See also: Infrastructure Architecture Requirements for Continuous Intelligence
E-commerce offers a great example in practice, leveraging fast data to make real-time recommendations based on internal and external events. A traditional batch-oriented approach would involve processing data and updating predictive models periodically (e.g., weekly or at best nightly) to determine a static set of recommendations that would be stored and used by applications until the next batch update. In contrast, retailers using fast data can pull from a customer’s current shopping cart, recent browsing history, similar purchases happening in the moment, real-time inventories, outside data like weather or breaking social trends, and more to make suggestions about what products they might need right now.
How leaders can drive fast data initiatives
While shifting to a fast data model is certainly a technological and architectural decision, it’s equally a business strategy consideration, simultaneously creating new opportunities and making existing processes obsolete. Successful implementation will require leadership, focus, and understanding of the following best practices.
#1 Switching from a ‘more data’ to a ‘fast data’ mindset
Making use of data as quickly as possible can open opportunities, but it requires a different mindset. Most organizations are trapped in a “more data is better” loop that prioritizes data collection over data action. Leaders need to think critically about their reasons for collecting more and more data and re-evaluate decisions based on immediacy. Asking “If I know what’s happening right now, what would I do differently?” helps leaders refocus on the most relevant information, saving on the time and cost involved in collecting more data. (It’s important to note this isn’t an either/or decision; some use cases may benefit from historical insight while others are best suited to immediate action.)
#2 Carefully evaluating technologies needed to support fast data
Switching to fast data requires updated systems that handle real-time data processing, and adding this infrastructure must not add undue complexity. Complexity is never a good thing in IT, but it’s particularly the enemy of fast data, slowing down data systems and processing, and making it difficult for internal teams to identify and take advantage of new data opportunities.
Ideally, fast data is integrated into the organization’s overall data management approach rather than added as yet another technology stack or silo. As part of a holistic data management strategy, businesses have the opportunity to consider shifting investments from historical repositories (such as data lakes, warehouses, etc.) to the technologies that can connect and process data in motion.
Consider data analytics, for example. Instead of using only predictive models to wring every last marginal return from a massive data lake, an organization might instead turn its attention and investment to fast analysis and immediate action.
#3 Ensuring new fast data initiatives can scale
Scalability is a two-fold challenge with fast data, impacting both technical and business process considerations. In both instances, teams too often start with a small-scale test environment, where demands are low and processes artificially contained, only to discover later that they don’t scale to meet real-world volume and velocity requirements.
Technical leadership must consider how the infrastructure can scale up and down as needed to facilitate fast iteration without disruption, as well as integrate with other initiatives such as mobile, IoT, etc. to meet future demands. Adopting a flexible and resilient cloud-native approach helps ensure that what works in the lab can scale to meet larger organizational requirements.
Similarly, business leadership must avoid creating “process or initiative silos” where fast data is limited to one aspect of the business (say, customer engagement) without considering how the data must impact other processes in the organization (billing, logistics, etc.). Business leaders can provide leadership on the overall strategy and help execution teams determine any potential growth areas that data initiatives will need to integrate and support.
It’s speed that counts, not volume
Fast data has many exciting implications across industries. By focusing on the fact that it’s not simply the volume of data that matters, but how quickly you can act on it, leaders can guide both their teams and the business overall to benefit from their data in new ways that both delight and retain customers, and accelerate their business forward.
