The algorithms businesses depend on to rate everything from credit scores to customer satisfaction may potentially be wrong – with nobody questioning it
Organizations are employing more and more analytic algorithms that provide real-time responses and reactions to situations and problems. Production systems can provide alerts or workarounds in response to sensor data, aircraft engines can operate at peak efficiency, and businesses can serve customers targeted offers or enticements as they visit sites or call in.
However, there’s a risk of relying too much on these algorithms without understanding the logic programmed within them. All too often, these systems may exacerbate biases that business leaders overlook, as these systems are branded as “untouchable” due to the fact they are based on pure math.
[ Related: Why Analytics Success Relies on System Design ]
But the math often is not pure.
That’s the argument put forth by Cathy O’Neil, mathematician, data scientist, and author of Weapons of Math Destruction: Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, speaking at the recent Strata conference in New York.
The ugly truth: analytic algorithms may be wrong
In her talk, O’Neil pointed out that the algorithms businesses depend on to score everything from credit scores to customer satisfaction levels may potentially be wrong – with nobody questioning it.
The truth is, many algorithms may be wrong, and thus reinforcing biases or erroneous information. They may even simply reflect the biases of their developers. “Algorithms are opinions embedded in code,” she said. In the process, she adds, “we’re hiding behind mathematics as a shield.”
With the rapid proliferation of real-time algorithms determining everything from creditworthiness to corporate performance, executives and managers need to become more intimately involved in designing AI and machine learning algorithms. Otherwise, decision-making gets wrapped up in unknown logic. “People who create algorithms embed their own definitions of success,” O’Neil said. “Machine learning doesn’t make things fair, it represents past patterns and automates those patterns.”
[ Related: Why You May Want a Career in Data Science ]
O’Neil described three issues stemming from the pervasiveness of algorithms:
- Widespread: Algorithms are making decisions about a lot of people.
- Secret: People don’t understand how they’re being scored.
- Destructive: Individuals may be unfairly denied access to resources because of biased algorithmic scoring.
Learn to question analytic algorithms
O’Neil also delivered a TED talk in April of this year that underscored the problems with algorithms. “Many algorithms represent a form of data laundering,” as O’Neil brands it.
“Algorithms don’t make things fair. They repeat our past practices, our patterns. They automate the status quo,” O’Neil said. She advises executives and managers to be more proactive in overseeing algorithm development and not to fear the math. “The bottom line is that algorithms are not sacred vessels not to be questioned. They should constantly be reviewed, and the logic behind them constantly questioned.”
