Artificial intelligence and big data have formed a truly symbiotic relationship, and they need each other to bring to fruition what both are promising.
If there are any lingering doubts that the fates of artificial intelligence and big data are intertwined, consider these recent quotes from two highly regarded thought leaders in this space:
“Throughout the business world, every company these days is basically in the data business, and they’re going to need AI to civilize and digest big data and make sense out of it.” (Kevin Kelly, co-founder of Wired)
“In the past, AI’s growth was stunted due to limited data sets, representative samples of data rather than real-time, real-life data and the inability to analyze massive amounts of data in seconds. Today, there’s real-time, always-available access to the data and tools that enable rapid analysis. This has propelled AI and machine learning and allowed the transition to a data-first approach. Our technology is now agile enough to access these colossal datasets to rapidly evolve AI and machine-learning applications.” (Bernard Marr, noted AI author and speaker)
AI and big data have formed a truly symbiotic relationship, and they need each other. Another well-regarded expert, Mike Manchett, senior analyst with Taneja Group, has been observing this revolution in the AI market, noting that a new generation of tools and platforms is helping to make AI a reality for businesses of all types and sizes. His observations are part of a special report on AI prepared in conjunction with the recent AI World conference in Boston.
Apache Spark is the Spark for AI Development. AI is a resource-intensive environment that gobbles up data by the truckload. While many enterprise infrastructures may not have been ready for this, open source tools make the proposition highly cost-effective and compelling. In particular, Manchett points to the widespread adoption of Apache Spark, which provides in-memory, real-time queries and fast machine learning at scale.
AI Democracy Approaches. Artificial intelligence is no longer the exclusive domain of PhDs. Now, thanks to a new generation of easier-to-use tools and platforms, tech professionals can start building and deploying AI solutions within their projects. “Today, I get a real sense that big data analytics is finally within reach of the average engineer or programming techie,” Manchett says. “There is now a large and growing middle ground where smart non-data scientists can be very productive with applied machine learning even on big and real-time data streams.” AI is not quite ready for non-technical business, users, however, Manchett cautions. “To achieve the big data and AI goals, you still need to understand extract, transform and load concepts and what machine learning is and can do, but you certainly don’t need to program low-level parallel linear algebra in MapReduce anymore.”
See also: How AI is Transforming Retail
Data Takes Center Stage: Data management long has been relegated to the backwaters of data centers, as something to be stored and secured. With the increasing emphasis on AI as a strategic tool, there’s a recognition that data is essential to the corporation’s future as well – and needs to managed as such. “Like other systems management areas, data flows can have their own service-level agreements, availability goals, performance targets, capacity shortfalls and security concerns,” Manchett says. “Flowing data has provenance, lineage, veracity and a whole lot of related metadata to track dynamically. This is a new world, and providing big data and big data flows with their own systems management focus has real merit as data grows larger and faster.”
All Data Becomes Big Data: “Eventually, all data will be big data, and machine learning — and the broader AI capabilities — will be applied everywhere to dynamically optimize just about everything,” says Manchett. “Given the power easily available to anyone through cloud computing, the impending explosion of IoT data sources and increasingly accessible packaged algorithms, the possibilities of big data and AI are becoming real in our lifetimes.” AI and machine learning, then, will become commonplace, with “the data center of the near future may soon be a converged host of all the data an organization can muster, continually fed by real-time data flows, supporting both transactional systems of record and opportunistic systems of engagement, and all driven by as much automated intelligence as possible.”
