How streaming analytics, the rise of data science and the growth of cloud could change the digital transformation path for enterprise in the coming months.
This year will be the when real-time big data analytics will come to the forefront of the enterprise. Although it has been said before, this year will see a convergence of several factors that will make this prediction a reality.
Companies are increasingly using cloud platforms and advanced data processing solutions to derive essential business insights to enhance operational processes, improve customer service and provide executives with critical data points. This market shift is driving the push to create more value from big data and investments in real-time analytics, and growing the need to use data science and machine learning for greater insight.
In the coming months, these five factors will enable enterprises to unlock the advanced power of their data:
#1: A shift to the cloud accelerates
Enterprises are steadily moving their on-premise IT and data processing to the public cloud. This trend is expected to accelerate through this year, driven by the growing availability of pre-built, reliable, scalable platforms-as-a-service (PaaS) for every possible application development and deployment need across the organization. Developers and everyday business users will use these cloud application platforms to design and operate applications, easier and faster, with minimal coding, while focusing on the core business logic.
Additionally, the main concerns related to security are diminishing, as the public cloud becomes more robust and secure. This is validated by the growing use of public and private cloud by traditionally cloud-shy conservative businesses like large financial services companies and banks, even for critical business processes. The total cost, complexity, and burden of trying to manage, scale and run large application devops on private infrastructure will only make the cost of public cloud services more and more attractive to enterprises.
#2: Real-time and big data analytics processes produce greater insights
Real-time analytics and stream processing will truly arrive in 2018. Owing to a large number of successful early adopters, proof-of-value, and proof-of-concept projects, enterprises will begin large-scale implementation of stream processing and advanced real-time analytics as part of their core data processing infrastructure.
See also: Why streaming analytics is critical to digital transformation
It will be driven by the key business objectives including competitive pressure, a growing need for fast data processing, the ability to act on business opportunities in real-time, and the demand for contextual and time-relevant customer experiences. To meet this demand, vendors will start offering vertical end-user applications like pre-built churn analytics, anomaly detection, predictive maintenance, recommendation engines and customer 360 frameworks on big data platforms.
There will be a shift to derive higher value from data lake investments. Transactional platforms will connect in real-time to big data lakes to enable faster, intelligent processing of data as it arrives. Direct business intelligence (BI) solutions running on top of the data lake providing scalable, fast interactive response to queries spanning very large data sets will get critical mass adoption.
#3: Apache Spark will continue its dominance
Apache Spark will remain the de-facto big data processing engine that will be used for both traditional ingest and ETL functionality, loading the data lake and machine learning training and prediction jobs. In terms of deployment, most enterprises will begin to leverage the “build once and deploy both as batch and streaming jobs” feature of Spark structured streaming APIs which makes it possible to run identical code in both modes with very minimal changes.
Spark deployments by experienced users will encompass a wider range of use cases while a rapidly increasing number of companies will implement Spark for the first time. Penetration levels will equal and could even surpass Hadoop adoption due to cloud-based approaches and non-Hadoop usage of Apache Spark.
And as these Spark implementations rise, there will be a greater demand for productivity tools and user interfaces to manage it and other big data jobs. Self-service functionality will enable a wide range of users to build big data and fast data analytic applications even without deep technical skills.
#4: Emerging streaming analytics will gain traction
Apache Flink will begin to rise as a “true” low-latency streaming analytics engine, filling in the space left from the fading of Apache Storm and the delayed arrival of millisecond level latency capability of Apache Spark which is still currently limited by its “micro-batch” paradigm. Kafka-Streams is likely to be the only real competitor to Flink, if any, as an event streaming engine, while Apex, Samza and some others will stay as small niche players in terms of total adoption in the market given the lack of wide communities contributing to and supporting them.
#5: Data science will be business’ biggest value driver
In the past year, we’ve seen a lot of conversations around machine learning, predictive and prescriptive analytics and artificial intelligence – and it will only continue to grow in 2018 as enterprises deploy these technologies. When companies see real use-cases with clear financial outcomes, technology and implementation details take a back seat. These success stories are typically driven by data science work that’s done on top of big data sources.
Data science “model management” as a mature and out of the box feature set will begin to appear more. There will be a rise in data science workbenches and self-service productivity tools for easier model building, training, and execution. Deep learning, which made a big entry in 2017 in the enterprise, will be introduced to many new use cases in 2018 especially as a solution to the problem of manual retraining of aging or low performing traditional machine learning models.
Machine learning, advanced predictive and prescriptive analytics and artificial intelligence (AI) scale will begin to expand in their enterprise usage and graduate from being early stage and niche solutions. Rule-based and manual “subject matter expertise-based” approaches to problems will transform into machine learning driven solutions e.g. predictive caching, predictive and prescriptive dev-ops or application performance management.
Data is a powerful corporate asset that executives wish to fully harness ever more. With the many investments in cloud migration, data lakes, in-memory computing, modern business intelligence and data science technologies, 2018 will be the year when a large number of enterprises will derive breakthrough value from these technologies and go through a transformation into future-ready data-driven real-time enterprises.
