DEV Community

NEXUS tools
NEXUS tools

Posted on

grep can't check the numbers in a Japanese LLM summary. Here is why, and a small fix.

#ai #llm #python #mcp

Say your agent summarizes Japanese meeting transcripts. You want one cheap guardrail: every number in the summary must also appear in the transcript. In English you could almost do that with a regex over the digits. In Japanese you can't, because the same value is written in many ways.

The transcript says 三千五百万円. The model writes 3,500万円. Same amount (35 million yen), zero shared digits. A few more pairs that a digit search misses:

Transcript Summary Same value?
三千五百万円 3,500万円 yes
令和7年4月1日 (Japanese era date) 2025/4/1 yes
午後2時 ("2 o'clock p.m.") 14:00 yes
1割5分 (old-style rate) 15% yes
問い合わせ30件 (30 inquiries) 30人 (30 people) no, wrong counter

The last row is the dangerous kind. The digits match, so a naive check passes, but the unit changed.

Normalize first, then compare

I wrote JP Number Grounding for this. It pulls every quantity, date and time out of both texts, converts them to one form (kanji numerals, 万/億, full-width digits, era years, 午前/午後, counters), and then checks the summary against the source.

Here is a real run. The source:

来期の予算は三千五百万円とし、令和7年4月1日から開始する。説明会は午後2時から。問い合わせは30件あった。次回は5月18日。
Enter fullscreen mode Exit fullscreen mode

A summary with two mistakes in it:

予算は3,500万円、2025/4/1開始。説明会は14:00。問い合わせ30人。期限は5月1日。
Enter fullscreen mode Exit fullscreen mode 
from jpnumground import check
r = check(source, summary)
r["ok"]  # False
Enter fullscreen mode Exit fullscreen mode
Fact in summary Normalized Status
3,500万円 35000000円 supported
2025/4/1 2025-04-01 supported
14:00 14:00 supported
30人 30人 unit_mismatch
5月1日 ????-05-01 unsupported

It caught both: the counter swap and a deadline (May 1) that nobody said. The meeting said the next session is May 18.

How it compares with a digit search

The repo ships 30 test cases. On them the tool flags the right cases 30/30, and a plain "are these digits in the source?" check gets 14/30. I wrote the cases and the rules, so read that as a regression suite, not a benchmark.

Using it from an agent

It is an MCP server, a CLI and a Python library. Python standard library only, no network calls, MIT.

{"mcpServers": {"jp-number-grounding": {"command": "python", "args": ["-m", "jpnumground.server"], "cwd": "/path/to/jp-number-grounding"}}}
Enter fullscreen mode Exit fullscreen mode

A rule that works well in a system prompt: after writing minutes or a summary from Japanese text, call jp_number_grounding_check and fix or drop every unsupported fact before sending.

What it does not do

It checks that values are present, not that they are used correctly. A summary can copy 15% from the source and attach it to the wrong item, and this tool will pass it. It only handles Japanese text, and it reports unknown number forms instead of guessing.

Code and the 30 cases: https://github.com/japan-ai-tools/jp-number-grounding

Top comments (0)

⌂ 换网址