RC RANDOM CHAOS

DAYJOB: Tax Reveals AI's Professional Judgment Gap

· via Hacker News

Original source

Dayjob: Tax – evaluating AI agents on 50 professional tax assignments

Hacker News →

Surge AI released DAYJOB: Tax, a benchmark designed to evaluate how well AI agents handle professional tax work. Rather than testing narrow technical knowledge, the benchmark simulates real-world scenarios where clients bring incomplete requests and tax professionals must identify overlooked issues. Two examples illustrate the challenge: an agent correctly evaluated two sale dates for a client’s condo but failed to mention a deadline that would eliminate her tax eligibility if she delayed past August 2027; another agent recommended asset division for a couple without verifying their actual marital status, which could determine whether the transaction triggers taxes and penalties.

The benchmark comprises 50 assignments spanning eight areas of tax practice—from individual returns to international taxation and corporate transactions. Each assignment includes up to 123 files with realistic documentation: filed returns, handwritten receipts, dispatch logs, cryptocurrency records, and correspondence. Tax experts with experience at firms like PwC, Deloitte, and EY created and reviewed the tasks to ensure they reflect genuine professional workflows. Agents are graded not just on calculation accuracy but on whether they recognize consequential issues, evaluate conflicting evidence, and identify what needs investigation before providing advice.

On the current leaderboard, Claude Sonnet 5.5 achieved a 10.8% pass rate at the 95% rubric threshold, followed by Claude Fable 5.1 and Claude Opus 5.5. The results highlight a gap between AI performance on defined tax questions and its ability to handle the full scope of professional judgment. Existing benchmarks measure whether models can answer specific tax questions correctly; DAYJOB tests whether they can recognize that clients may not know what question to ask.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.