Study highlights the need for racial diversity cues in AI training data to enhance trust

A recent study by researchers at Penn State University explores the impact of racial diversity cues in artificial intelligence (AI) systems and suggests that transparent data can foster trust and perceived fairness among users. The study delves into how displaying details about the racial makeup of training data and the backgrounds of the individuals who label it can influence users’ perceptions of AI systems, which have been shown to harbour systematic biases related to race and other characteristics.

AI systems, including home assistants and chatbots like ChatGPT, are widely used for a variety of applications, but their effectiveness is directly linked to the data on which they are trained. These systems can inadvertently perpetuate biased decision-making if their training data lacks diversity. Users often remain unaware of these biases, which can manifest themselves once the AI has been utilised, potentially causing harm without users having pre-emptive knowledge of the bias embedded within the system.

The research, detailed in the journal Human-Computer Interaction, involved experiments led by S. Shyam Sundar from the Center for Socially Responsible Artificial Intelligence at Penn State, alongside Cheng “Chris” Chen, an assistant professor of communication design at Elon University. The core idea was to investigate whether communicating the racial composition of AI training data and labelers could alter users’ expectations and trust in AI fairness.

During the study, participants were presented with two scenarios reflecting different compositions of AI training data. In one scenario, the training data was shown to be equally comprised of images from three racial groups: white, Black, and Asian. Labelers were also equally diverse. In the other scenario, the data and labelers were predominantly from one racial group, accounting for 92% of the total. Participants interacted with a facial expression classification AI tool, HireMe, which evaluated candidates’ employability based on their expressions. Notably, half the participants experienced a racially biased AI system, manipulated to favour one racial group over others, while the other half engaged with an unbiased system distributing assessments equally among the racial groups.

Key findings from the research indicated that showing racial diversity in training data and among labelers increased participants’ trust in the AI system. Furthermore, the ability for users to provide feedback enhanced their sense of agency, hence encouraging potential future use of the AI. Interestingly, the study also discovered that providing feedback was seen as unnecessary by some participants when the system appeared unbiased—specifically among white participants who perceived it as an extra burden.

The researchers underscored the importance of a concept known as the “representativeness heuristic,” where individuals are more inclined to believe in the inclusivity and fairness of an AI model if its racial data representation aligns with their understanding of diversity. Professor Sundar highlighted the criticality of diverse representation in both the training data and among the people who label it. He cautioned that systems trained with predominantly one racial perspective might misinterpret expressions belonging to other races, potentially skewing outcomes in contexts such as emotion detection.

This research recommends that the origins and composition of AI training data should be made accessible to users, fostering transparency and accountability. Even when users do not directly access these details, their availability is posited to signify ethical practice, a step towards reinforcing fairness and trust in AI systems.

Source: Noah Wire Services