Technology & AI

Ai2 releases StateBench to test AI on changing long documents

The benchmark measures whether language models can follow information that changes across lengthy texts, including in frontier models.

By Standard Topic. AI-generated from the reporting listed under Coverage. Published . Updated .

This story is no longer being updated. It reflects reporting up to .

What happened

Ai2 has introduced StateBench, a new benchmark that tests how language models track changing information across long documents. Separately, one report says AI can retrieve the right document yet answer from a wrong, similar record nearby.

Timeline

  1. Nov 3, 12:15 PM ET: StateBench, a benchmark for testing how language models track state changes over long documents, was published.
  2. Nov 4, 4:15 AM ET: Reports said a study found AI can use the wrong document despite retrieval, as similar records nearby can reduce accuracy.
  3. Nov 4, 5:38 AM ET: A legal AI test reported that correctly extracted values rose from 52% to 70% after revisions in driving-under-the-influence cases.

Coverage

More from Technology & AI

All Technology & AI news · Technology & AI story archive