Welfare Systems Have Been Monitored by Algorithms for Years. Now They Face Criticism for Partiality
Human rights organizations have initiated a fresh legal challenge regarding the use of algorithms for identifying errors and fraud in France’s welfare system, asserting that single mothers are unfairly impacted.
Today, a coalition of human rights organizations initiated legal proceedings against the French government regarding its utilization of algorithms to identify erroneous welfare payments, claiming that they are discriminatory towards disabled individuals and single mothers.
The algorithm, in use since the 2010s, breaches European privacy regulations and French anti-discrimination legislation, claim the 15 organizations participating in the case, which include digital rights organization La Quadrature du Net, Amnesty International, and Collectif Changer de Cap, a French group advocating against inequality.
“This marks the first instance of a public algorithm facing a legal challenge in France,” states Valérie Pras from Collectif Changer de Cap, expressing her desire for a ban on such algorithms. “Various social organizations in France employ scoring algorithms to focus on the disadvantaged.” If we manage to have [this] algorithm prohibited, the same will hold true for the others.
According to a lawsuit submitted to France’s highest administrative court on October 15, the French welfare organization, the CNAF, examines the personal information of over 30 million individuals—those receiving government assistance, alongside their household members and relatives.
Utilizing individual personal details, the algorithm assigns a score ranging from 0 to 1 to each person, reflecting how likely it predicts they are to receive payments for which they are not qualified—either through fraud or error.
France is among several nations employing algorithms to detect errors or fraud within its welfare system. The previous year, a three-part investigation by Lighthouse Reports examined fraud-detection algorithms within European welfare systems, specifically looking at their application in the Netherlands, Denmark, and Serbia.
Individuals with elevated risk scores may then experience what welfare recipients throughout the bloc have termed as stressful and invasive investigations, which can also result in their welfare payments being halted.
“The operations carried out by the CNAF represent extensive monitoring and an excessive infringement on privacy rights,” the legal papers regarding the French algorithm state. “The impact of this algorithmic processing specifically influences the most vulnerable individuals.”
The CNAF has not disclosed the source code of the model it is presently employing to identify erroneous welfare payments. However, according to an analysis of earlier iterations of the algorithm, believed to be operational until 2020, La Quadrature du Net asserts that the model unfairly targets marginalized populations by assessing individuals with disabilities, for instance, as higher risk than others.
“According to Bastien Le Querrec, a legal expert at La Quadrature du Net, individuals obtaining a social allowance intended for those with disabilities [the Allocation Adulte Handicapé, or AAH] are specifically affected by a variable in the algorithm.” “The risk score for individuals on AAH who are employed is raised.”
Since it assigns higher scores to single-parent families compared to two-parent families, the organizations claim it indirectly discriminates against single mothers, who are statistically more inclined to be primary caregivers. “Le Querrec states that in the criteria for the 2014 version of the algorithm, the score for beneficiaries divorced for under 18 months is elevated.”
Changer de Cap affirme avoir été contacté par des mères célibataires et des personnes handicapées en quête d’aide, après avoir fait l’objet d’une enquête.
The CNAF agency, responsible for distributing financial assistance such as housing, disability, and child benefits, did not promptly reply to a request for comment or an inquiry about whether the algorithm in use has notably changed since the 2014 version.
Similar to France, human rights organizations in various European nations contend that they place the lowest-income individuals under heavy surveillance—often resulting in serious outcomes.
When tens of thousands of individuals in the Netherlands—mostly from the nation’s Ghanaian demographic—were wrongly accused of cheating the child benefits system, they were not only instructed to return the funds that the algorithm claimed they supposedly took. Numerous individuals assert that they were similarly burdened with overwhelming debt and ruined credit scores.
The issue lies not in the design of the algorithm itself, but in how it is implemented within the welfare system, states Soizic Pénicaud, a lecturer on AI policy at Sciences Po Paris, who has prior experience with the French government regarding the transparency of public sector algorithms. “Implementing algorithms in social policy carries significantly greater risks than advantages,” she states. “I have not encountered any instance in Europe or globally where these systems have yielded positive outcomes.”
The case has implications that extend beyond France. Welfare algorithms are anticipated to serve as an initial examination of how the EU’s new AI regulations will be applied when they become active in February 2025. From that point forward, “social scoring”—the application of AI systems to assess individuals’ behavior and subsequently impose harmful treatment on some—will be prohibited throughout the bloc. “Several welfare programs that engage in fraud detection could, in my view, represent social scoring in action,” states Matthias Spielkamp, cofounder of the nonprofit Algorithm Watch. However, representatives from the public sector are expected to contest that definition, as debates over how to characterize these systems are likely to result in legal disputes. “I believe this is quite a difficult question,” remarks Spielkamp.
