Data not Guesswork: How to Handle Bias in AI

As artificial intelligence becomes commonplace in industries ranging from finance to medicine to defense, a revolution is underway. Research labs in both industry and academia have devoted immense resources to combating bias — observed favoritism for or against a certain group — in AI. This year, for example, Google doubled its AI ethics research staff size. Not only is AI fairness a popular subject of research, but it is also entering the regulatory domain— the U.S. Equal Employment Opportunity Commission (EEOC) recently launched an initiative to ensure algorithms comply with federal civil rights law.  

But what exactly is fair AI? Researchers have varying definitions of fairness with different aims and motivations. Current research has created tool kits for reducing bias in AI, but these tools often employ subjective notions of bias that introduce personal value judgements into AI algorithms. Ultimately, AI researchers ought to ensure fairness by focusing on concrete, statistical notions of bias instead of normative ones. Further, efforts to ensure fairness should be driven by researchers, not bureaucratic and technically inept government agencies.

AI algorithms model the distribution of data that they are trained on. Obviously, when the data is not representative, a number of problems will arise. For example, facial recognition algorithms trained on datasets composed almost solely of white individuals will perform poorly on people of color — not a good outcome. Oftentimes, the most effective solution to this problem involves ensuring that training datasets are representative of the entire population, thus mitigating discrepancies in accuracy.

However, an influential paper, Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings, suggests that using balanced data is not enough. This work discusses word embeddings — an algorithm where words are spaced closer or further apart according to the similarity of their meanings. It addresses the fact that common algorithms produce embeddings for the word “engineer” that are much closer to “man” than to “woman.” The paper posits that while these embeddings “could capture useful statistics [about the real world]” it is “better to err on the side of neutrality.” The work ultimately suggests researchers should manually realign the original embeddings to eliminate these learned statistics.

This view on neutrality, however, raises a number of difficult subjective questions. If we manually shift statistics that our model has discovered, we have no objective way of deciding the extent to which we should make a change. Should the word vectors for all words be equidistant from those for “man” and for “woman?” When should they be closer to “woman?” Or to “man?” Additionally, if we realign vectors with respect to gender, for example, we risk unintentionally introducing new biases on other axes like race or sexuality. Granting individuals the authority to manually make these final decisions gives them immense power and subverts the fundamental values of neutrality.

Ultimately, assuming one’s model is representative of the real world, using the unadjusted results could actually improve both fairness and accuracy. Not only can the statistics learned by AI models improve performance, but models that are representative of the real population are, by definition, those with the least statistical bias. Thus, efforts to ensure fairness and neutrality should aim to eliminate instances of true statistical inaccuracies or misrepresentations. In particular, such instances include cases in which the statistics learned by the model may inaccurately describe the real world or the model may be more accurate for members of one group than for members of another.

One such case involves Amazon’s hiring algorithm which, having been trained mostly on the resumes of men, picked up on meaningless signals that were biased against women. Specifically, the model learned to correlate arbitrary words that happened to appear more frequently in the resumes of men with successful candidates, a clear statistical inaccuracy. Once such biases have been discovered, researchers should focus on building balanced datasets to mitigate these issues. By focusing their efforts on the dataset layer instead of the final trained model, researchers can make principled decisions that still guarantee a degree of separation between their choices and the resultant AI system, thus incentivizing neutrality.

Unfortunately, even well-meaning solutions following this basic framework can be counterproductive if the details are implemented poorly. When the Apple credit card launched in 2019, for instance, its algorithm offered smaller credit lines to women than to men. The issuing bank, Goldman Sachs, was puzzled at how the algorithm could be biased when it did not use gender as an input and had been vetted by a third-party for fairness.

It turned out that the algorithm was using inputs that happen to correlate with gender, biasing it against women without even explicitly considering gender. By manually adjusting the dataset, Goldman had just hidden the bias instead of actually resolving it. Thus, the individuals and institutions responsible for making specific choices about the implementation of these algorithms will dramatically affect their end results.

Concerningly, ownership for these choices is being increasingly placed in the hands of government agencies. The Federal Trade Commision (FTC), the main regulatory body in charge of algorithmic fairness, recently sent stern warning to companies in a post entitled Aiming for truth, fairness, and equity in your company’s use of AI, and, just this year, the National Science Foundation allocated $20 million to research AI fairness.

Senator Ron Wyden proposed the Algorithmic Accountability Act, which would require companies to ensure unbiased — by the government’s definition — algorithms, under penalty of law. The state of California is considering its own law, which would require companies to submit an algorithmic bias report to California Department of Financial Protection and Innovation each year. It is unclear if these plans would hire staff qualified to examine AI models, or if they would rely primarily on existing government bureaucrats.

Given the government’s spotty record at even understanding modern technology, let alone regulating it, placing such an ethically complex and technically challenging problem in the hands of a bureaucratic and unelected body could spell disaster for the technology industry. Ultimately, neither the subjective whims of nontechnical government officials nor personal views of researchers should be used to alter algorithms. We must focus on specific and statistical notions of fairness rather than introducing our own views into algorithms.