Avoiding bias in supervised machine learning tools

United Kingdom

On 11 December, the FCA published a research note on bias in artificial intelligence (AI). This publication is part of a series from the FCA exploring how AI intersects with financial services. The document does not constitute guidance for firms but aims to stimulate discussion on issues involved in building and implementing AI systems.

The research note focusses on bias in supervised machine learning models. Supervised machine learning models predict outcomes (such as the future risk of someone making an insurance claim) based on past training data. The FCA focusses on how unjustified differences in predictions or outcomes might arise for three groups of customers with particular characteristics: (1) protected characteristics, (2) vulnerability and (3) other demographic characteristics (such as region, occupation or income). Data bias was found to be one of the top five perceived current risks of AI in the BoE and FCA’s recent survey on AI.

What is bias?

In this context, bias refers to unjustified differences in predictions or decision-making based on the demographic characteristics, or wider life or social circumstances of a person (such as characteristics of vulnerability). These disparities may arise from several sources, including historical biases in data, modelling choices or how humans selectively use predictive models. The models treat data correlations as purely mathematical relationships and miss the broader societal context of why these patterns exist in the first place. So, for example, a premium calculation model that incorporates occupation data that results in higher premiums being charged to those who work part-time may disproportionately affect those with health conditions who cannot work full-time. Another example would be a model which heavily weighs postcode in risk assessment and results in higher premiums being charged to applicants from minority ethnic groups simply because they live in certain postcodes.

What contributes to bias?

Bias can be introduced into an algorithm during the problem-framing, data collection and algorithm-building steps. During the problem-framing step, seemingly unrelated characteristics might be close proxies for demographic characteristics or characteristics of vulnerability and different input sources could introduce bias into the algorithm. At the data collection stage, data can lead to some groups being underrepresented or overrepresented which can lead to worse outcomes for some groups. Each stage of the model training then has the potential to introduce or perpetuate bias from the way data is collected, processed, weighted or used.

What are the key takeaways from the FCA’s research note?

The FCA is focussed on ensuring firms can responsibly and safely adopt AI and clearly identify risks and mitigate them.  It has recently highlighted concerns over firms using AI technologies in a way that embeds or amplifies bias, and which might lead to worse outcomes for some groups of customers. This research note is a contribution to the debate on bias in supervised machine learning to help firms identify considerations in the development of models and first-line mitigation strategies.

The key takeaways from the research note are:

  • Data issues arising from past decision-making, historical practices of exclusion, and sampling issues are the main potential source of bias.
  • Biases can also arise due to choices made during the AI modelling process itself, such as what variables are included, what specific statistical model is used, and how humans choose to use and interpret predictive models. 
  • In reviewing technical methods for identifying and mitigating such biases, these methods should be supplemented by careful consideration of context and human review processes. However, technical mitigation strategies may affect model accuracy and could have unintended consequences for model bias on other groups.

The FCA suggests that firms consider the following questions for supervised machine learning model construction and deployment to help measure and mitigate bias:

  • What demographic characteristic or characteristics of vulnerability could be studied?
  • What metrics could be used to measure any bias?
  • Is it feasible and appropriate to collect or proxy data on demographic characteristics or characteristics of vulnerability on an individual level, either for some or all customers?
  • Is it feasible to mitigate biases and, if so, how?
  • If demographic characteristics or characteristics of vulnerability cannot be measured or proxied, are there any realistic alternatives?

If you would like to discuss the adoption of AI in your business, please do not hesitate to get in touch with the contacts listed or your usual contacts at CMS.