How AI bias and adversarial attacks impact the fairness and safety of self-driving cars

In San Francisco, a self-driving car (SDC) ran over and dragged a pedestrian 20 feet before finally coming to a halt. There have also been numerous reported cases of driverless cars driving through active crime scenes. Yet another blocked the path of an ambulance rushing to the scene of a mass shooting. In these vehicles, actions and decisions are powered by the AI-driven control system, often trained on datasets that may not represent all environmental variations (such as urban vs rural settings and an array of weather conditions) and pedestrian demographic variations (such as age, gender, and race). These gaps can result in SDCs posing major risks to passengers, pedestrians, motorists, and infrastructure.

While the benefits of SDCs are well known, such as less traffic congestion, and fewer human driver errors, their current limitations are less so. In this article we draw from the whitepaper, ‘From Data to Decisions: Bias, Fairness, and Security in AI for Self-Driving Cars,’ authored by Nada Madkour, Alice Nicoleta Pavaloiu, and Nikita Lukianets, exploring how bias and security vulnerabilities can compromise fairness and safety in SDCs and the role of transparency in mitigating those risks.

Bias in AI systems

AI bias refers to the disparities and flaws within AI systems that can lead to unfair outcomes for certain communities or groups. These biases have surfaced in tools ranging from hiring software and facial recognition software to self-driving car pedestrian detection and the COMPAS recidivism tool.

While AI systems are trained on vast amounts of data, those datasets are often not as diverse or representative of their deployment environments as they should be. Bias can be introduced in numerous forms into AI systems, resulting in real-world consequences. As Open Ethics researcher Nada Madkour shared in an article about self-driving car ethics, “the SDC’s ability to accurately decide on which lives to save and which to sacrifice very much depends on its ability to detect each and every one of those lives to begin with.”

Making sense of the many types of AI bias can be greatly assisted by classifying the different biases. Researchers suggest categorizing AI bias by where the bias originated from: data to algorithm bias, algorithm to user bias, and user to data bias.

Data to model bias

Data-to-algorithm bias, also known as model bias, stems from the data used to train machine learning (ML) models. Within this category there are numerous types, including:

Measurement bias: resulting from the misuse of particular features. In other words, using the wrong tools or features to measure something, which creates a wrong result. For example, the COMPAS recidivism tool, which used arrest rates as a proxy for criminal behavior, even though arrest rates themselves reflect biased policing patterns rather than actual crime rates.
Omitted variable bias: is a result of having variables missing from a training dataset. In other words, leaving out important data, so the model makes a wrong guess. For example, a hiring algorithm trained without accounting for career gaps due to parental leave, causing it to unfairly penalize candidates for a pattern it didn’t have the context to understand.
Representation bias: can occur when the collected population sample lacks diversity. For example, when a facial recognition system is trained mostly on lighter-skinned faces, causing it to perform less accurately for darker-skinned individuals who were underrepresented in the training data.This type of bias can also occur in unsupervised learning models as they are trained on unlabeled data and can easily misjudge underrepresented or overrepresented data.
Sampling bias: occurs as a result of unbalanced or unfair data. For example, when a health app’s fitness data is collected primarily from users of a single fitness-tracker brand, skewing conclusions toward the habits of a specific socio-economic group rather than the general population.
Aggregation bias: occurs when false conclusions are made about individuals from observing an entire population. For example, when a medical AI model is trained on population-wide data showing a certain diabetes risk threshold, then applies that same threshold to individual patients from ethnic groups whose risk factors differ from the overall population average, leading to missed or inaccurate diagnoses for those individuals.

Beyond that particular classification framework, bias can also be introduced during the data annotation process.

Annotation bias

Annotation bias occurs when human data labelers introduce systematic errors, personal prejudices, or cultural blind spots into training data. Researchers suggest that annotation bias can take the form of:

Entity bias: a result of certain entities appearing more frequently than others.
Labeler bias: the influence of existing biases that data labelers have. This can also include a type of bias introduced by the mental fatigue of a data labeler.

One of the concerns with AI bias is that it can form a feedback loop where AI models “learn” certain biases from human interactions, then the biased AI output influences human judgment, resulting in a more prominent bias in subsequent decisions, creating a bias feedback loop. This cycle not only perpetuates the unfair treatment of certain population groups, but it can deepen systemic inequalities, resulting in real-world problems that are magnified over time.

In the whitepaper, the authors introduce a potential method for evaluating and understanding bias and fairness called the Fairness-Bias Matrix. Access the full whitepaper to learn more.

These biases, along with adversarial attacks, can introduce uncertainty and errors into an SDC’s decision-making. With incomplete or incorrect data, that decision could result in serious or even fatal human harm.

Image credit: Anton Grabolle / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/

AI biases and adversarial attacks: risk vectors in self-driving cars

In SDCs, it’s not only biases in AI systems that can create risk. Adversarial attacks are another potential threat vector that can skew an SDC’s decisions. Both can jeopardize the physical safety of the vehicle’s occupants and pedestrians.

Several studies have revealed how bias in SDC ML can lead to unfair and even potentially lethal outcomes. According to researchers, SDC pedestrian detection systems have exhibited gender, racial, and age bias. In one study, the researcher found that 72% of the methods examined had a significantly higher miss rate on female pedestrians. Other studies also found disparity in miss rates to the detriment of dark-skinned, elderly, and child pedestrians.

In 2018, a self-driving Uber vehicle crashed into and killed a woman in Tempe, Arizona. It was reported that the system design did not account for jaywalking and failed to classify Elaine Herzberg as a pedestrian, resulting in her death. Another recent study that evaluated eight pedestrian detection algorithms with four common SDC training datasets found that on average these models have a 7-10% higher dark-skinned pedestrian miss rate and a 16-19% higher children miss rate compared to lighter-skinned and adult pedestrians.

Another vector that can affect black-box type AI models is adversarial machine learning (AML). AML is composed of a set of techniques that allow an adversary to exploit an ML system by manipulating its data, model, or both. The impact of AML attacks may result in the model learning the wrong things or making wrong decisions.

Adversarial attacks on fairness have shown how easily hackers can trick an AI model into making unfair, racially biased decisions by changing a small portion of its training data. One study showed that when AI models don’t have strict fairness rules, hackers can easily target minority groups and disrupt the system’s fairness. Similar research revealed how hackers could secretly trick an AI into becoming biased against a specific gender, while overall model accuracy was not affected, creating an undetectable bias.

AI transparency aims to shed light on how AI systems work and make decisions. In the case of AI bias and adversarial attacks, a lack of transparency is a common and dangerous thread in both.

AI Towards fairness and safety in SDCs with AI transparency

To address the AI biases, we may need something other than a one-size-fits-all approach. One study introduces the concept of bias management rather than trying to remove bias altogether. The research proposes a transparency-based strategy that allows users to make more informed choices.

Another notable methodology suggests including data labeler characteristics and background information with the dataset. These two methods suggest that implementing transparency mechanisms throughout the ML training process can address labeler bias, helping identify and mitigate biases at early stages of the lifecycle and prevent bias from escalating during the ML lifecycle.

Transparency in AI systems can also play a role in defending against adversarial attacks on SDCs. When a model’s decision-making process is interpretable, security researchers and engineers can more easily identify when an input has been manipulated, since the system’s reasoning for a given output becomes visible rather than hidden inside a black box. Improving the explainability and transparency in deep learning for autonomous vehicles could be a promising direction for advancing detection and security against adversarial attacks, researchers have posited.

We hope that models designed for interpretability will be better equipped to flag suspicious or manipulated inputs before they lead to unsafe decisions, empowering end users and building more confidence in the reliability of SDC systems.

At Open Ethics Initiative, we propose a solution called the Open Ethics Data Passport as a transparency-by-design approach to reduce AI bias in self-driving cars.

Image credit: Grendelkhan, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons

The Open Ethics Data Passport

The Open Ethics Data Passport (OEDP) provides insight into the origins of AI models, where the dataset originated from, the data type (synthetic or human-generated), the labeling practices, and labelers’ profiles.

It is an iterative tool built to provide in-depth transparency and awareness and can help tackle issues of bias that lead to unfair outcomes in any AI use case, including SDCs. It is suitable for use on any AI product and its components throughout the product stages of development, including in the early stage of system design.

The OEDP is like a nutrition label for AI. It labels the contents or constituent parts so they’re no longer a mystery. The ultimate aim is for a deeper level of transparency, where the ingredient, in this case the AI product, is broken down through disclosure of sub-ingredients or the AI product components. This provides a powerful level of insight and transparency that aids in identifying systematic bias in trained AI models.

Bias in self-driving car AI reaches beyond skewed data and underrepresentation. It can also be introduced or magnified through adversarial attacks, making it as much a safety and security concern as an ethical one. Addressing these risks, and building tools like the Fairness-Bias Matrix to understand them and the Open Ethics Data Passport to mitigate them, will be essential to the safe, large-scale deployment of self-driving cars.

If you’d like to learn more about bias, fairness, and security in AI for self-driving cars, access the full whitepaper down below.

From Data to Decisions Bias, Fairness, and Security in AI for Self-Driving Cars