The Netherlands has become a testing ground for AI in social services. An algorithm played a central role in the childcare benefits scandal that falsely accused tens of thousands of families of fraud and triggered the government’s resignation. In Rotterdam, our investigation revealed an algorithm that ranked welfare recipients based on clothing and Dutch language proficiency, disproportionately targeting single mothers with migrant backgrounds.
So when Amsterdam—one of Europe’s most progressive cities—told us in 2023 that it was developing a “fair” algorithm to detect welfare fraud, we were intrigued.
For five years, Amsterdam’s welfare department ran a high-stakes experiment guided by Responsible AI: a framework of technical and ethical principles intended to ensure fairness, transparency, and accountability in automated systems. Promoted by academics, NGOs, multinational institutions, and a growing consulting industry, Responsible AI has emerged as a key response to scandals surrounding algorithmic decision-making.
Yet few systems built under these principles have been scrutinized independently. Over several years, Amsterdam spent hundreds of thousands of euros, hired consultants, consulted academics, rigorously audited the system for bias, and even invited welfare recipients to provide feedback on its design.
The city also allowed Lighthouse to observe the process.
Despite this effort, the system failed. In collaboration with MIT Technology Review and Trouw, we sought to understand why. We gained unprecedented access to the system, the officials who built it, and the critics who challenged it.
The result is one of the first in-depth examinations of a system developed under Responsible AI guidelines—and what happens when its promises meet reality.
STORYLINES
Together with MIT Technology Review and Trouw, we investigated a fundamental question: what does it truly mean to deploy an algorithm fairly?
Previous reporting, including our own, has often highlighted the worst-case scenarios—poorly designed or intentionally discriminatory systems. These accounts typically sidestep the more difficult questions of when, if ever, such technology should be deployed and what fair AI should actually look like.
Amsterdam followed the Responsible AI playbook closely. It debiased the system after early tests revealed ethnic bias, engaged academics and consultants, and ultimately opted for an explainable algorithm over a more opaque one. The city even convened a council of welfare recipients—the very people the system would evaluate—who criticized the project sharply.
Yet in real-world deployment, the pilot system continued to exhibit biases. It was also no more effective than the human caseworkers it aimed to replace. Under mounting political pressure, officials ultimately ended the project, quietly concluding a costly, multi-year experiment.
We detail the lessons drawn from Amsterdam’s attempt to build a Responsible AI system, highlighting the disagreements over whether such frameworks can fulfill their promises—or whether some AI applications are fundamentally incompatible with human rights.
METHODS
In 2023, months after our original Suspicion Machines investigation, one reporter filed a public records request with Amsterdam for code and documents related to its welfare fraud detection system.
Unlike previous investigations, where agencies delayed or obstructed access for months or years, the city disclosed everything immediately and invited an online meeting. It became clear why: the city had gone to great lengths to design a fair system and believed it had succeeded.
At the time, the city was preparing a pilot where the model would score real-world welfare applicants. The pilot failed: the model was biased and ineffective.
By fall 2024, nearly a year after our reporting began, the city shelved the project entirely. To understand how biases had emerged during model training, reweighting, and deployment, we needed to audit the system—but faced a roadblock. Although the city shared code and documentation, Europe’s GDPR prevented sharing data on how the system had scored actual recipients—crucial information for auditing.
In what we believe is a first, the city cooperated via remote access: we sent code and tests to officials, who ran them on real data and returned aggregate results. Lighthouse has published our methodology online, along with the underlying code and data on GitHub. Months of reporting—including interviews with officials, welfare recipients, and experts—allowed us to reconstruct how the project unfolded and why it ultimately failed.