Qiniso · localecheck

How often do AI agents get international data wrong?

Frontier LLMs confidently produce plausible-but-wrong answers for everyday non-US data — dates, tax rates, currency, phone numbers, holidays, addresses. They rarely signal the error, so it surfaces downstream as a missed flight, a rejected invoice, or a failed KYC check. localecheck measures the gap and fills it.

⚠️ Illustrative figures below. The numbers shown are placeholders from the harness in --mock mode to demonstrate the format. Run the open-source eval against a real model to populate this page with live results — see “Reproduce it” at the bottom.

Error rate by category

Each category is scored against an authoritative source (libphonenumber, Unicode CLDR, python-holidays, and curated tax records). “Error” means the model disagreed with the verified answer.

Category	What breaks	Error rate*
Currency formatting	US-style format applied to other locales	~40–65%
Date parsing	Reads DD/MM as MM/DD; guesses impossible dates	~30–45%
VAT by date	Applies today’s rate to historical / temporary periods	~25–40%
Address parsing	Mis-splits non-US postcode/city ordering	~25–35%
Public holidays	Misses regional bank holidays; over-calls famous ones	~10–20%
Phone validation	Fails to reject invalid numbers	~5–15%

*Ranges are illustrative pending a real-model run. Phone is near-solved by existing libraries; tax-by-date and currency formatting are the highest-value gaps.

Concrete failures (real LLM-style errors)

Move my flight to 03/04 — user is in London (en-GB)

✗ Agent rebooks 4 March → ✓ Correct: 3 April

Generate a UK invoice dated 2009-06-01

✗ Agent applies 20% VAT → ✓ Correct: 15% (temporary financial-crisis rate)

Show revenue to a German reader: 1234.5 EUR

✗ €1,234.50 (reads as ~1,200) → ✓ 1.234,50 €

“Delivery in 2 business days” quoted on 19 Apr 2025 (England)

✗ Counts Easter Monday as a working day → ✓ 21 Apr is a bank holiday

Reproduce it

The benchmark is open source. Point it at any model and it regenerates a report:

pip install -r requirements.txt && python generate_dataset.py
export ANTHROPIC_API_KEY=… && python run_eval.py --provider anthropic

It writes a per-category error table and the full list of failures. Run it against each new model release to see whether the gap is closing or durable.

Fix it in one tool call

localecheck is an MCP server (and TypeScript library) that returns verified answers and flags impossible inputs instead of guessing. Add it to Claude Desktop, Cursor, or any MCP client with npx -y localecheck — it's live on npm and listed in the official MCP Registry.