RiddleBench Read paper
Findings of EACL 2026 Dataset available

Reasoning is easy.
Until the clues move.

RiddleBench is a new benchmark of 1,737 puzzles built to test whether large language models can sustain a coherent mental model—not merely recognize a pattern.

Deepon Halder · Alan Saji · Thanmay Jayakumar · Ratish Puduppully · Anoop Kunchukuttan · Raj Dabre

Explore the dataset Read the paper
1,737
curated puzzles
10
leading LLMs
69.26%
best overall score
RQ3 / ROBUSTNESS TEST INTERACTIVE

Seating arrangement

Does clue order matter?

Qwen QwQ 32B 17.66%

Nine people sit in one row, all facing north.

  1. 01Cheenu sits exactly in the middle.
  2. 02No person sits to the right of Ishita.
  3. 03Dona is fourth to the right of Faria.
  4. 04Gaurav and Harish sit next to each other.
  5. 05Anu is second to the right of Harish.
Reported category accuracy Original constraints
baseline

The logical content stays identical. The score change is reported in Table 4.

Explore the benchmark
01

The premise

A right answer is not enough.
The reasoning must survive contact with change.

Many benchmarks reward the final answer. RiddleBench stresses the machinery behind it: multi-step deduction, spatial awareness, constraint satisfaction, and the ability to reject a plausible—but wrong—line of thought.

Why it matters

Systems that cannot audit their own reasoning may compound errors as they deliberate.

02

Inside the benchmark

1,737 puzzles.
Four modes of thought.

Sourced from competitive examination material, OCR-extracted, structured, and then manually verified by the authors.

Dataset composition

Constraint-heavy by design, with a deliberate mix of linear, spatial, relational, and symbolic tasks.

60%

Sequential reasoning

Establish a linear order from interlocking rules.

25%

Seating arrangements

Maintain a spatial layout while constraints accumulate.

8%

Blood relations

Infer kinship from a chain of stated relationships.

7%

Coding–decoding

Discover and apply transformations to symbols and words.

EvaluationZero-shot
Temperature0.7
Thinking budget8,192 tokens
Dataset licenseCC0
03

Model performance

The leaderboard has
no comfortable winner.

Every evaluated model leaves a substantial portion unsolved. Switch the reasoning mode to see how dramatically each model's profile changes.

Correct answers across all 1,737 puzzles.

Scores = correct answers (%) Source: Table 2 · RiddleBench
04

Failure signatures

Models don't just get lost.
They defend the wrong turn.

The paper moves beyond accuracy to diagnose how errors propagate, harden, and resist correction.

01 / HALLUCINATION CASCADE
45.2% incorrectly validated flawed peer reasoning

When given DeepSeek-R1's flawed trace, Qwen QwQ 32B often accepted the plausible premise instead of rebuilding the solution from scratch.

02 / SELF-CONFIRMATION
17.3% successfully caught its own flawed logic

Self-review performed far worse than peer review. The model failed to identify its own error in 67.7% of trials.

03 / ERROR FIXATION
4.4% reversed an incorrect judgment on another pass

Iterative re-evaluation barely broke the cascade. Once committed, the model usually entrenched the error.

RQ3 / PROMPT PERTURBATIONS

Same logic.
Different surface.
Lower score.

Reordering constraints or inserting one irrelevant sentence should not change the answer. For Qwen QwQ 32B, it often changed performance.

−7 pp0+3 pp
Blood relationsconstraints shuffled
−6.70
Coding–decodingirrelevant sentence
−3.87
Seatingconstraints shuffled
−3.69
Seatingirrelevant sentence
−3.08
Blood relationsirrelevant sentence
+2.74

Change in percentage points · Table 4

05

From document to diagnostic

Built by machine.
Checked by people.

Automation made collection possible; manual verification made the benchmark trustworthy.

01

Source

Public mock-exam PDFs containing questions, answers, and reasoning traces.

02

Extract

Gemini 2.5 Flash OCR digitized the source material.

03

Structure

Raw output was separated into clean, consistent puzzle records.

04

Verify

Every data point was manually checked for transcription and correctness.

OPEN BENCHMARK · CC0 DATASET

Put reasoning
under pressure.

Use RiddleBench to test not only whether a model arrives at the answer, but whether its reasoning remains dependable on the way there.

Open on Hugging Face ACL Anthology
Halder et al. “RiddleBench: A New Generative Reasoning Benchmark for LLMs.” Findings of EACL, 2026.
Copied to clipboard