Sampling bias
Machine learning models are trained on a large amount of historical data to make predictions. When the sample data fed into the machine learning model is itself tainted, it can lead to erroneous outputs. For instance, take the case of a hiring algorithm Amazon developed in 2014 to screen resumes. This algorithm was trained on resumes from the pool of existing software engineers at Amazon, which was overwhelmingly male. As a result of this sampling bias, the hiring tool systematically discriminated against female applicants.
Protected attribute proxies
Protected attributes are personal features, such as race, color, age, and sexual orientation, which are protected by law and cannot form the basis of making a decision. However, ML models can inadvertently cause discrimination by using proxies that correlate with these protected attributes. For example, imagine building a credit decision model for a country where people of one race live in the northern part and people of another race live in the southern part. Now, even though race isn’t explicitly fed into the model, it might still discriminate based on race simply by virtue of where a person lives.
Selection bias
Sometimes enough representative data isn’t collected while training an ML model, which creates an incomplete picture of a real-world scenario. Due to this selection bias, the ML algorithm is likely to perform well with the specific data set it has been trained on but quite poorly with a more general data set. For example, consider an ML model that has been trained on high net worth individuals to predict the attrition rate of customers from a bank. As a result of this selection bias, the algorithm will probably produce bad results for lower net worth individuals, who have substantially different characteristics than high net worth individuals.
Fairness in machine learning models
The concept of fairness is difficult to capture since we first have to debate what decisions are fair and what aren’t. For instance, if you’re at a party that’s serving pizza, how do you fairly distribute the pizza slices? It’s probably fair if everyone at the party gets an equal number of slices. But don’t you think the people who made the pizza should get more slices since they put in more effort than the others? Or maybe it’s fair to give more pizza slices to the hungrier people at the party? This subjectivity in what’s fair is precisely why fairness can be difficult to define. And that is why it is as much a business problem as it is a technical one.
One way to define fairness in machine learning would be an attempt to correct the algorithmic bias in ML models arising from the way training data is sampled, aggregated, labeled, and processed. Some of the criteria used to make ML models fairer are:
- Anti-classification: This is a simple fairness criterion that states that a model should not use protected attributes while making predictions. But such a model is still susceptible to bias due to proxies, and hence this is a weak fairness criterion.
- Group fairness: Group fairness requires that the probability of a model predicting a positive outcome should be similar for subpopulations distinguished by protected attributes.
- Equalized odds: As per this criterion, an ML model’s predictions must have the same true positive and false positive rates among different subgroups. That means that the rate at which the model makes mistakes should be equal across different subgroups.
Apart from setting up fairness criteria, we can use multiple strategies to improve fairness at different stages of the machine learning pipeline. For instance, with model evaluation and auditing, we periodically evaluate a model’s fairness and bias in data. If there’s any suspicion or evidence of unwanted bias, we can investigate further and debug the source of the bias. Some other strategies include model inference, model training, and feature engineering.
