Evaluating & Ranking GPT-5 Reasoning Ability

Given the proliferation of reasoning models, we wanted to go beyond knowledge-based benchmarks to test reasoning abilities such as pattern recognition, lateral thinking, abstraction, contextual reasoning (accounting for British cultural references), and multi-step inference.

In addition to reasoning, we aimed to assess how effectively models make decisions when presented with judgment calls—such as choosing between making an educated guess based on available clues or calling a function to retrieve additional information. This capability is crucial for building multi-agent orchestration systems.

Another objective was to measure improvements in the latest GPT-5 models, particularly using the reasoning effort and verbosity parameters, comparing them to previous iterations and evaluating their token and reasoning time efficiency.

What is Only Connect?

Only Connect tests contestants' ability to identify connections between seemingly unrelated clues. It prioritizes lateral thinking, pattern recognition, and creative problem-solving over quick recall. The game consists of four rounds:

• Connections: Players identify the common thread linking 1-4 clues
• Sequences: Players predict the fourth element in a sequence after seeing 1-3 clues
• Wall: Players group 16 elements into four categories (similar to the NYT Connections game)
• Missing Vowels: Players reconstruct phrases with removed vowels and spaces using cryptic clues

Given its emphasis on clever reasoning rather than knowledge recall, Only Connect provides an ideal challenge for benchmarking LLMs' reasoning capabilities. We also wanted to track performance improvements across successive model generations.

Methodology

We selected models for analysis including GPT-3, GPT-4-Mini, GPT-4.1, Claude Sonnet 4, Opus 4, and Opus 4.1, along with GPT-5 using eight different parameter configurations (low/high verbosity and minimal/low/medium/high reasoning).

Questions were sourced from the Only Connect game show, following official rules. For rounds requiring straight guesses, we provided LLMs with all available clues and used structured output parameters to receive JSON responses. When contestants could request additional clues, we implemented this as a function call triggering another API call with the supplementary information.

For evaluation, deterministic answers (such as missing vowels) were assessed using standard string methods. For questions with multiple correct answers or varying phrasings, we employed the deepeval library for nuanced evaluation. Points were assigned using the game's official scoring system. We simulated eight randomly selected episodes from series 3-10 and aggregated the results.

Results Overview

The best-performing models were GPT-5 and similar reasoning-optimized models. Verbosity had minimal impact on accuracy.

We found a strong correlation between response time and accuracy. GPT-5 models with higher reasoning parameters (high, medium) consistently outperformed those with lower settings (low, minimal).

Similarly, when analyzing token usage, reasoning models consumed comparatively high token counts but demonstrated strong effectiveness. The verbosity parameter significantly affected token usage with only minor impacts on accuracy.

While GPT-5 and other reasoning models perform well, they come at a cost in both time and token usage.

Regarding individual rounds, models performed best on Missing Vowels—unsurprising given this round prioritizes speed over lateral logic. LLMs routinely handle poor grammar and spelling, making this relatively straightforward. The Wall round proved most challenging, with significant performance gaps between top and bottom performers. This likely stems from the prompt complexity containing 16 different elements, which more powerful reasoning models processed more effectively. We'll compare NYT Connections games against the Only Connect dataset in a future post.

Next Steps

We'll publish the complete dataset this week alongside a granular analysis identifying which questions posed the greatest challenges for models. We'll also implement a more realistic competitive format, pairing models against each other and allowing points for correctly answering questions opponents miss.

Interested in pushing the boundaries of AI research and knowledge? Reach out to careers@ingram.tech.