Sage Nord
What 86,726 broken code samples tell us about trusting AI-generated code
Most conversations about AI-assisted coding are qualitative. Someone says a model "hallucinates" or "makes silly mistakes," and everyone nods, but nobody attaches a number to it. A study accepted for the 37th IEEE International Symposium on Software Reliability Engineering (ISSRE 2026), by Rodrigo Pato Nogueira, Marco Vieira, and João R. Campos of the University of Coimbra's CISUC research center, puts real numbers behind the intuition.
The researchers ran seven large language models across four compiled languages: C++, Java, C, and Rust. From that, they built a corpus of 86,726 code samples that failed to run cleanly. Of those, 52,669 samples failed to compile outright, and 34,057 threw at least one runtime error during unit testing. Rust produced the largest share of compilation failures (23,416), followed by C (12,304), C++ (9,953), and Java (6,996).
What makes the study useful, rather than just alarming, is the breakdown by cause. The team classified errors by root cause using an LLM classifier, validated the classifications manually, and then compared patterns across languages. The result: failure modes are not universal, they are language-specific. "Missing import" errors account for 56.6% of all C++ compilation failures, but only 8.1% of Java compilation failures. A model that is unreliable in one language for one reason can be unreliable in a completely different language for a completely different reason.
The most practically important finding, though, is about self-correction. When the researchers gave models feedback and a chance to fix their own mistakes, the fix rates were inconsistent and, in absolute terms, low. Missing-import errors were fully resolved only 12.4% of the time. Undeclared-variable errors were fixed just 5.3% of the time. In other words, even with an explicit second attempt, models did not reliably clean up after themselves, and which errors they could fix varied by error type, not by any predictable pattern engineers could easily anticipate. The paper also notes that generated code frequently omits basic input validation or memory-safety checks, the kind of gap that turns a compile-time nuisance into a production security issue.
For a consultancy working across iOS, Android, web, backend, and AI, this lines up with something we already build into how we work: AI-assisted code generation is a productivity tool, not a substitute for review. This study gives that instinct a number. If a model fixes missing-import errors only about one time in eight when explicitly asked to self-correct, and the failure patterns shift depending on the language you're writing in, then human review, static analysis, and language-specific tooling are not optional extras. They are the actual reliability layer.
The practical takeaway is not "don't use AI for code." It's "don't assume the model catches its own mistakes, and don't assume its mistakes look the same in Rust as they do in Java." Build review and testing pipelines around that reality, not around the hope that the next model version makes it unnecessary.