Artificial intelligence, machine learning, and other algorithm-driven technologies have the potential to revolutionize eLearning through their ability to parse data, identify patterns, categorize information, and predict outcomes. Algorithms enhance the efficiency and personalization of many types of online training, performance support, and problem-solving tools. However, any machine learning or data-crunching algorithm is only as reliable as its coding and the data used to “train” it, and built-in bias in algorithms is becoming a significant concern in technologies used for tasks as disparate as making parole decisions, determining the terms of consumer loans, deciding which job applicants to interview, guiding students on college and career paths—or predicting which entry-level employees might succeed as managerial candidates.
Learning Solutions offers this overview of bias in algorithms as part of a deeper exploration of the potential of artificial intelligence to transform eLearning design. A companion article will examine proposals for evaluating algorithms or platforms to detect or mitigate bias.
Encoded bias
Some algorithms are built to be biased: In a paper on auditing algorithms, Christian Sandvig and three co-authors describe SABRE, an early algorithm for booking airline flights that American Airlines created in the 1950s. Executives were open about—even proud of—their algorithm, with the company president declaring in testimony to Congress that “biasing SABRE’s search results to the advantage of his own company was in fact his primary aim.” Sandvig notes that today, what he calls “algorithm transparency” remains a pressing problem—and it can be much harder to detect. Algorithms might still be programmed to favor the products or services of their creator—Amazon and Google, among others, have been accused of this.
Some bias is unintentional. Algorithms might use a combination of demographic characteristics to determine what products, services, or opportunities to offer to individuals. While the characteristics used superficially seem unbiased, the result could be discriminatory. For example, consumers might be offered different investment opportunities or credit cards based on a combination of where they live, whether they rent or own their homes, and their income. Those characteristics can offer strong hints as to the individuals’ race or ethnicity and patterns or decisions based on them could result in minorities being disproportionally steered to less favorable products.
In an eLearning context, algorithms might limit the course selection shown to some learners or exclude valuable content from a curated site, based on coded criteria that the L&D team are unaware of.
Data sets used to “train” machine learning algorithms
Machine learning is at the heart of much personalized and adaptive eLearning. Algorithm-powered machine learning-based technologies are “trained” using a data set. Bias can taint the algorithm when the data set used to train the algorithm is not sufficiently diverse or representative.
Facial recognition algorithms, for instance, learn to recognize faces by being taught to identify faces from thousands or millions of images. Eventually, the algorithm “learns” what elements to look for in an image to decide whether the image is a human face, a dog, or a cat. The machine learning detects as its pattern that a human face has specific features—nose, eyes, lips—and that these features follow a pattern of shapes, sizes, etc. If the training data set is not diverse, the pattern will tend to match only a limited subset of the shapes and features that human faces include. In well-publicized examples, facial recognition programs have had trouble recognizing Asian and African or African American faces because the vast majority of the images in the training sets contained mostly Caucasian faces.
A similar pattern-matching algorithm could learn other discriminatory patterns. Imagine, for example, a career-counseling or coaching tool or an algorithm intended to predict which new hires were the most promising candidates for management training. The training data set might logically use historical employment records—an accurate set of data representing employees who had become successful managers—to learn the pattern of what characteristics those managers shared. Based on that pattern, the algorithm could predict which applicants or new hires might succeed as managerial trainees. However, if the data set reflects the current and historical makeup of management in American corporations, the vast majority of examples of successful managers would be white men. Thus the tool would learn an unintended and extremely undesirable pattern: The best management candidates are white men. Similar biases have been found in algorithms that, for example, present men with ads for high-paying and executive-level positions far more frequently than they present those job ads to equally qualified women, or that guide women and minorities to different academic or career paths than white males with similar grades, course histories, and other characteristics.
An additional area where bias in data used to teach algorithms could affect corporate eLearning is in the algorithms and technologies that use natural language recognition and processing. This technology—that powers machine translation, searches and interactions in digital assistants, and much more—uses types of AI that could be widely implemented in eLearning and performance support tools.
Natural language use in AI is built upon systems that “teach” programs to associate word pairs. These associations, called “embeddings,” are learned by the machine algorithms from “thousands of articles the algorithms had automatically scavenged and analyzed from online sources such as Google News and Wikipedia,” according to Ben Dickson of TechTalks. The machines scour this content to learn how people use language, with the goal of creating sentences, responses, and examples that will sound natural to users of the program.
But the articles reflect human and social bias. Articles about engineers and high-tech workers, for example, are more likely to include male names than female names, since these fields are male-dominated. But, while a human might notice this imbalance and see a social problem, all the algorithm sees is a pattern, a pattern that it builds into its language processing rules. Thus when a team of Boston University and Microsoft researchers studied Word2vec, a widely used algorithm, they found extreme gender bias in many of the associations.
As some of these examples show, bias in algorithms or their results can result from perfectly reasonable choices as to which parameters are used, unintentional selection of data sets that are insufficiently diverse, or use of data and tools that reflect historical and current realities, with all of their inequalities.
Whether bias is embedded in the basic code at the foundation of an AI- or algorithm-based technology or creeps in as a result of training data, it has the potential to become widespread. The increasing use of off-the-shelf code or code libraries disseminates this biased code globally and to an enormous variety of applications.
“Given the convenience and reusability of libraries, over time many programmers use popular code libraries to do common tasks. If the code being reused has bias, this bias will be reproduced by whoever uses the code for a new project or research endeavor,” Joy Buolamwini, the founder of the Algorithmic Justice League, wrote in Medium. “Unfortunately, reused code at times reflects the lack of inclusion in the tech space in non-obvious but important ways.”
The bias in AI algorithms will also be present in any eLearning built upon these flawed technologies or platforms. Awareness is the first step toward avoiding bias; in a companion article, Learning Solutions will present suggestions for auditing algorithms and platforms to detect bias. And The eLearning Guild is presenting a Data & Analytics Summit August 22 – 23, 2018. Register now for the opportunity to learn more about the relationships between data and content.