RC RANDOM CHAOS

AI Benchmarks Miss 90% of US Working Hours

· via Hacker News

Original source

90% of US working hours are spent on tasks not covered well by AI benchmarks

Hacker News →

Mercor mapped 29 existing AI benchmarks against the US Department of Labor’s occupational taxonomy (923 occupations, 18,797 tasks) and found that 90% of American working hours fall on tasks with little or no benchmark coverage. Only 208 of the 923 occupations have a single task represented by at least 5 benchmark prompts, and 6 of 22 job families have none. This makes it difficult to assess how well current models handle the work most Americans actually perform.

Coverage is not evenly distributed. Benchmarks concentrate on the largest labor markets and the tasks rated most exposed to AI automation, which are roughly 6 times more likely to appear in benchmark prompts than unexposed tasks. Within that, tasks with higher domain-expert importance scores are substantially overrepresented. A concrete gap: lawyers’ tasks are heavily benchmarked while legal secretaries and court clerks, similarly AI-exposed, are barely covered. The long tail of smaller, middle-income knowledge-work occupations is underrepresented.

The authors note the results are provisional, based on an initial 29-benchmark list that deliberately caps entries per job family to maximize breadth. They plan to expand the index and weight model performance by task-level working-time shares to produce occupation-level capability estimates that enterprises, policymakers, and workers can actually use.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.