Innitor Logo
Benchmarking GPT 4.5
TechnicalBy Ruilin Xu3 min read

Benchmarking GPT 4.5

AIChatGPTGPTTechnicalOverview

Introduction

ChatGPT 4.5, positioned as a significant mid-generation upgrade by OpenAI, demonstrates marked improvements in knowledge, reasoning, and emotional intelligence over its predecessors. This analysis compares ChatGPT 4.5 with OpenAI's earlier models O1 and O3, and Anthropic's Claude 3.7 Sonnet across various standard benchmarks and real-world applicability.

Key Findings

Language Understanding (MMLU & GPQA)

  • Multitask QA (MMLU): GPT-4.5 achieves ~85% accuracy, up from ~81% for GPT-4o. Claude 3.7 slightly outperforms it, achieving ~86-89%.
  • Domain-specific QA (GPQA): GPT-4.5 significantly improves to ~71%, surpassing GPT-4o’s ~54%, though still trailing O3-mini (~80%) and Claude (~78-85%).
  • Multilingual performance: GPT-4.5 notably outperforms GPT-4o across multiple languages, indicating superior multilingual integration.

Mathematical & Logical Reasoning

  • Advanced Math (AIME): GPT-4.5 significantly improved to 36.7%, compared to GPT-4o’s 9.3%, yet still far behind O3-mini’s exceptional 87.3%.
  • Logical Reasoning & Puzzles: Claude 3.7’s "extended thinking" mode notably surpasses GPT-4.5 on complex logical reasoning tasks (Claude: 21/28 correct; GPT-4.5 lagging).

Coding and Software Benchmarks

  • HumanEval: GPT-4.5 achieves ~76-77%, a substantial improvement from GPT-4o (~67%), yet behind Claude 3.7's leading performance (~82-83%).
  • SWE-Bench Verified: GPT-4.5 solves 38% of tasks, lagging significantly behind O3-mini (61%) and Claude 3.7 (~62%).
  • Complex Coding Tasks: Claude 3.7 excels in multi-file, tool-assisted coding scenarios, outperforming GPT-4.5.

Factual Accuracy and Hallucination

  • GPT-4.5 dramatically reduces hallucination, with only a 19% rate on PersonQA compared to GPT-4’s previous 52%.
  • Claude 3.7 also emphasizes accuracy, notably improving nuanced harm-distinction, though with less explicit focus on hallucination metrics.

Real-world Applications

  • Both GPT-4.5 and Claude 3.7 handle extended contexts (~128K tokens), essential for tasks requiring long document analysis or multi-turn dialogues.
  • Claude 3.7’s extended thinking provides significant advantages for structured reasoning, agentic workflows, and complex multi-step tasks.
  • GPT-4.5 emphasizes a natural conversational flow and context-aware interactions, providing superior "emotional intelligence."

Strengths and Weaknesses of ChatGPT 4.5

Strengths

  • General-purpose excellence: Strong across diverse tasks, particularly factual accuracy and multilingual understanding.
  • Reduced hallucinations: Significantly improved reliability in knowledge recall.
  • Natural interaction: Enhanced conversational abilities, more intuitively following user intent.
  • Integration and versatility: Robust support for file uploads, image inputs, and web browsing.

Weaknesses

  • Coding-intensive tasks: Falls short of specialized models like O3-mini and Claude 3.7.
  • Extreme reasoning tasks: Lacks Claude's dynamic "think longer" reasoning capabilities for exceptionally complex problems.
  • Benchmark increments: Modest improvements on established benchmarks, questioning its revolutionary status versus iterative refinement.

Unique Differentiators

  • ChatGPT 4.5: A balanced, versatile AI, excelling in general knowledge and user-friendly interaction.
  • OpenAI O3-mini: Specialized in STEM tasks, significantly outperforming others in coding and math.
  • Claude 3.7 Sonnet: Offers unique extended thinking capabilities, superior for complex coding tasks and detailed logical reasoning.

Implications

ChatGPT 4.5 reinforces OpenAI’s lead in general-purpose AI models, excelling in conversational ability, factual accuracy, and broad task coverage. However, for specialized STEM or intricate coding applications, O3-mini and Claude 3.7 currently provide stronger solutions.

Conclusion

ChatGPT 4.5 represents a substantial refinement of the GPT-4 architecture, making meaningful strides in accuracy, reduced hallucination, and interaction quality. While not universally dominant across all benchmarks, its balanced strengths position it as an exceptional all-around AI assistant. Future iterations will benefit from incorporating features inspired by specialized competitors to further enhance performance across all domains.

References