Ai2 releases StateBench to test AI on changing long documents
The benchmark measures whether language models can follow information that changes across lengthy texts, including in frontier models.
By Standard Topic. AI-generated from the reporting listed under Coverage. Published . Updated .
This story is no longer being updated. It reflects reporting up to .
What happened
Ai2 has introduced StateBench, a new benchmark that tests how language models track changing information across long documents. Separately, one report says AI can retrieve the right document yet answer from a wrong, similar record nearby.
Timeline
- Nov 3, 12:15 PM ET: StateBench, a benchmark for testing how language models track state changes over long documents, was published.
- Nov 4, 4:15 AM ET: Reports said a study found AI can use the wrong document despite retrieval, as similar records nearby can reduce accuracy.
- Nov 4, 5:38 AM ET: A legal AI test reported that correctly extracted values rose from 52% to 70% after revisions in driving-under-the-influence cases.
Coverage
- Legal AI Test Records 70% Correct Value Extraction After Revisions (tokenpost.com)
- StateBench Benchmark Reveals Limitations in AI’s Long-Document Comprehension (thechenabtimes.com)
- Study finds AI can use wrong document despite retrieval (itbrief.co.uk)
More from Technology & AI
- China puts humanoid robots to work serving food and sorting parcels
- Newsom proposes sweeping AI safety rules for Washington
- Google releases Gemini 4 Neon AI model with million-token context
- Microsoft and Intel widen their cloud chip partnership
- Google DeepMind rolls out Genie 4 to select subscribers
- AI helped AMD shorten chip design work, OpenAI says