Dutch welfare ruling spotlights AI redlining risk in insurance

A Dutch court ordered the Netherlands government to halt use of a welfare-fraud detection algorithm after finding that it breached human rights protections, Fortune reported. The ruling matters beyond welfare policy because governments and companies are using machine-learning systems to judge risk in areas from benefits enforcement to insurance pricing.

The system, known in English as System Risk Indicator, or SyRI, was used by four Dutch cities to flag benefits applications for closer review, according to Fortune. It combined information from 17 government databases, including tax files, vehicle registrations and land registries.

Fortune reported that the cities did not screen every applicant with SyRI. They used it in poorer neighborhoods with large immigrant populations, including residents from Muslim-majority countries.

The court said SyRI violated the “right to private life” under European human rights law, according to Fortune. The court also warned that the system could discriminate on the basis of socio-economic status, ethnicity or religion, and said its use did not appear to fit with Europe’s GDPR privacy rules.

The decision came from a district court and could be appealed, Fortune reported. Even so, the case is being watched as a potential marker for how European authorities treat automated risk scoring by public agencies.

Insurance raises similar questions

Fortune’s Jeremy Kahn reported that the insurance industry is increasingly using machine learning to improve underwriting. That shift has prompted concern among consumer and privacy advocates that automated pricing could create “digital redlining,” in which disadvantaged groups face worse terms because of data patterns tied to income, neighborhood, ethnicity or other traits.

Daniel Schreiber, co-founder and chief executive of New York-based insurance startup Lemonade, told Fortune that machine learning could widen access to financial services and lower costs if companies use it carefully. He also acknowledged the risk that poor design could produce unfair outcomes.

Schreiber told Fortune that insurers should test fairness through what he called a “uniform loss ratio.” Under that approach, the share of claims paid out compared with premiums collected should remain consistent across race, gender, sexual orientation, religion and ethnicity if underwriting is fair.

He argued that insurers should collect information tied to actual risk rather than protected identity. Fortune reported that Schreiber used the example of candle use in the home: asking whether a customer lights candles may relate to fire risk, while asking about religious affiliation would be discriminatory.

Schreiber said insurers may need more customer data, not less, to make those distinctions, according to Fortune. He said public mistrust, privacy scandals such as Cambridge Analytica and regulators’ limited understanding of machine learning have pushed sentiment in the opposite direction.

Pricing risk versus access

Kahn raised concerns that more precise risk pricing could leave some people unable to obtain coverage. He compared the issue to health insurance systems in which companies are allowed to avoid customers with pre-existing conditions.

Fortune also noted that people in poorer neighborhoods may pay more for home insurance because those areas can have higher crime or fire risk, even when residents have limited control over where they live. U.S. law bars policies with a disparate impact on protected classes unless a company can show a legitimate business necessity, Fortune reported.

Schreiber told Fortune that governments could address affordability separately by requiring higher premiums from wealthier households or areas and using the extra money to subsidize coverage in poorer neighborhoods. He said that question was distinct from whether the underwriting model itself treats risk fairly.

This story draws on original reporting from Fortune.