AI Coding: The 2025 Performance Review

The world of artificial intelligence is moving at a dizzying pace, but for those of us who’ve seen a few tech cycles come and go, genuine surprise is a rare commodity.

Yet, a seasoned journalist and programmer found himself genuinely astonished two years ago when OpenAI’s ChatGPT effortlessly whipped up a working WordPress plugin for his wife’s e-commerce site.

That moment, he recounts, ignited a deep dive into the unpredictable, often baffling, and sometimes brilliant realm of AI-assisted programming.

After subjecting no fewer than 14 large language models (LLMs) to a battery of four real-world coding tests, the verdict is in: the landscape has shifted dramatically, and the top contenders for your coding dollar (or lack thereof) might not be who you expect.

While some AIs have leaped from abysmal failure to stellar performance, others remain stubbornly unreliable, and in a few head-scratching instances, the free version outshines its costly counterpart.

The core truth emerging from this rigorous two-year examination is stark: not all chatbots are created equal when it comes to crafting code.

Even now, a significant portion—four out of the thirteen tested LLMs—struggle to produce functional plugins, let alone tackle more complex programming challenges.

This isn’t about writing entire applications; the consensus remains that AIs aren’t there yet.

Their forte lies in generating a few lines of code, debugging existing snippets, or providing quick solutions to specific problems.

Think of them as incredibly powerful, albeit sometimes eccentric, coding sidekicks.

So, who made the cut for 2025’s coding elite?

The list of recommended tools now stands at five, with a few free-tier honorable mentions that offer surprising value.

Leading the pack, and passing all tests, is ChatGPT Plus, powered by GPT-4o.

At $20 a month, it delivers solid coding results and even boasts a dedicated Mac application, a boon for developers juggling multiple screens.

However, it’s not without its quirks; the journalist noted an instance where it offered a dual-choice answer, one of which was incorrect, hinting at the lingering need for human oversight.

A close second, Perplexity Pro, also priced at $20 monthly, impressed by acing all tests and offering the unique ability to run multiple LLMs, including GPT-4o, Claude 3.5 Sonnet, and Llama 3.1.

This allows for a kind of “AI-driven code review,” cross-referencing output across different models.

Its primary drawback? An email-only login system devoid of multi-factor authentication, a surprising security oversight in an otherwise robust tool.

Perhaps the most astonishing turnaround story belongs to Google’s Gemini Pro 2.5 and Microsoft Copilot.

In previous tests, both were dismal failures.

Gemini, once a coding embarrassment, now passes all tests with flying colors, showcasing a “stunningly capable” improvement.

The catch for free users is severe query throttling, pushing users towards a token-based payment model that makes predicting expenses a guessing game.

Microsoft Copilot’s transformation is equally dramatic.

Previously deemed “astonishingly bad,” the free version of Copilot now passes all four tests, proving Microsoft’s commitment to learning from its mistakes.

This makes it a compelling, no-cost option for basic AI coding assistance.

Then there’s the truly baffling case of Claude 4 Sonnet.

This free version of Anthropic’s Claude model passed every single test, while its paid counterpart, Claude 4 Opus (which can cost anywhere from $20 to $250 a month), inexplicably failed half of them.

It’s a paradox that underscores the unpredictable nature of AI development, where more expensive doesn’t always mean better.

Beyond the perfect scorers, several free options offer significant utility.

Grok, Elon Musk’s AI integrated with X, surprised many by passing three out of four tests, hinting at potential future advancements given its lineage from Tesla and SpaceX.

Free versions of ChatGPT and Perplexity, while subject to throttling and limited to GPT-3.5, still perform admirably for day-to-day tasks.

Even DeepSeek V3, a Chinese LLM, earned its spot by passing three tests, outperforming established players like Google Gemini and Microsoft Copilot in previous iterations.

However, the field is also littered with cautionary tales.

DeepSeek R1, despite its hype, showed inconsistent coding quality.

GitHub Copilot, while seamlessly integrated into VS Code, often generates “very wrong” code, posing a significant risk if developers blindly incorporate its suggestions.

Meta AI and Meta Code Llama, Facebook’s offerings, consistently failed most tests, generating user interfaces with “zero functionality” or choking on simple challenges.

The consistent message from the journalist: don’t risk your programming projects with these tools until they show marked improvement.

The broader implications of this ongoing AI revolution for developers are clear.

AI isn’t a silver bullet for building entire applications, but it’s an increasingly indispensable tool for specific, granular tasks.

The rapid improvements seen in just two years suggest that today’s underdog could be tomorrow’s champion.

This constant state of flux means developers need to stay agile, testing and re-evaluating their AI assistants regularly.

And despite the incredible capabilities, the ZDNET-Aberdeen research indicates that only 8% of Americans would pay extra for AI, suggesting a strong preference for free or low-cost solutions among the general public.

In this wild west of AI development, consistency remains elusive, and the line between groundbreaking utility and frustrating hallucination is often blurred.

But for those willing to navigate its peculiarities, AI promises to remain a powerful, evolving ally in the programmer’s toolkit.

The journey of exploration continues, with future updates promised as this “warp speed” innovation continues to redefine the boundaries of what’s possible.