Recommendation engines aren’t just for Netflix and Amazon.
Name of Organization: American Express Corporation
Industry: Financial services
Location: New York, NY, USA
Business Opportunity or Challenge:
American Express Corporation is awash in data – multi-petabytes’ worth, reflecting more than 25 percent of US credit card spending, handling more than one trillion dollars in transactions annually. The financial services giant recognized, however, that it needed greater speed and transparency in the data it was managing. Corporate decision makers needed the power of real-time recommendation engines to better keep up with changing customer behaviors and methods of access, such as mobile technology.
The company’s pre-existing database structure was not ready for such a shift. As observed by
Ellen Friedman, big data consultant at MapR, American Express recognized “that traditional databases would not be enough to effectively handle the level of data and analytics needed for their projects.” Data volume keeps increasing, data sources are constantly changing, and more customers are accessing services via mobile devices. The company is tasked with keeping up with constantly shifting markets and increasing volume. “Part of that involves making a huge number of decisions, millions every day,” Friedman says.
To move into the realm of a real-time recommendation engine and visibility into customer behavior, the financial services company needed to build greater intelligence into the data that was moving through its enterprise systems. The nature of the company gives it the opportunity to see data from both the customer and merchant side of business—from millions of sellers and millions of buyers. While American Express “is never short of is data, the question is, how can they best leverage this data to improve the decisions they make?” Friedman relates. “If American Express can become just a little bit smarter in these decisions, it can have a huge advantage to customers and to the company.”
How This Business Opportunity or Challenge Was Met:
American Express moved to a big data infrastructure, built on a Hadoop platform and supporting machine learning to add intelligence to the data being gathered and processed. The company implemented the MapR data platform, which is designed to store, stream and facilitate search on data. The solution includes a real-time read/write file system, integrated NoSQL database, and a large array of Hadoop ecosystem tools intended to meet the needs of large-scale machine learning applications. MapR is engineered to make consistent snapshots for data versioning and to use remote mirroring for applications synchronized across multiple data centers.
The financial services firm employed machine-learning techniques across a wide range of interactions to better understand customer behavior. As part of this effort, the company implemented recommendation engine systems capable of processing and responding to the large volumes of data it handles, delivering predictions based on ever-evolving user histories.
“The beauty of this type of design is that the computational heavy lifting in which the learning algorithm is used in training the model can be done ahead of time, offline,” says Friedman. “Then conventional techniques such as a search engine can be put to work to easily deploy the system, making it able to deliver rapid online recommendations in real-time.”
The new system also was designed to handle large-scale queries fast. Real-time response is needed when validating credit card transactions, which requires querying “millions of records and returning a response in a second or less,” Friedman states.
Measurable/Quantifiable and “Soft” Benefits:
American Express is employing the technology to acquire new customers, about half of whom come from online. The real-time recommendation engine also provides value-added services to customers and merchants.
American Express built a “machine-learning mobile phone application to provide customized recommendations for restaurant choices,” Friedman says. “When the customer gives permission, the machine learning application uses recent spending histories and card member profile data for a huge number of transactions to train a recommendation model. The model predicts which restaurants a particular customer might enjoy and makes those recommendations.” This service not only benefits customers, but also provides feedback to restaurant merchants on the attractiveness of their offers.
The machine-learning implementation has also played a key role in fraud detection and prevention at American Express. Inputs such as card membership information, spending details, and merchant information, are pattern-matched against evolving algorithms in real time to flag transactions that have a high probability of being fraudulent. The pressure is on, as “a customer has swiped their card to make a purchase and expects to get approval immediately,” Friedman states. The large-scale machine learning deployment is adding far greater precision to the systems’ predictive capabilities.
(Source: Ellen Friedman, Apache Drill and Apache Mahout Committer, Big Data Consultant at MapR)
Want more? Check out our most-read content:
Research from Gartner: Real-Time Analytics with the Internet of Things
Frontiers in Artificial Intelligence for the IoT: White Paper
Data Visualization: How a Futures Exchange Sees Clearly
Video: Three Analytics Companies Explain Approaches
Three Types of IoT Analytics: Approaches and Use Cases
Liked this article? Share it with your colleagues!
