Can an AI Coding Agent Find Useful C++ Optimizations?

Experiment Architecture

The experiment pipeline combines deterministic program analysis with AI-generated code candidates. Each stage is independently verifiable, and the pipeline rejects candidates that fail any gate.

UTFCPP v4.0.6 — Primary Case Study

October 24, 2026 · Source revision b26a5f718f4f370af1852a0d5c6ae8fa031ba7d0

What Happened

The model proposed bypassing a redundant scalar-validity predicate after UTF-16 surrogate checking, using the same internal append implementation. The candidate passed all correctness, sanitizer, and reference checks. However, the measured performance showed no statistically supported improvement. The confidence interval spans both sides of 1.0, so neither a gain nor a regression can be established.

Validation Workload

Protected multilingual roundtrip workload (148,488 bytes, 32 full passes), independently generated Python-codec/FNV reference, 71 Unicode golden/boundary cases plus malformed UTF-8/UTF-16 checks, and the unmodified upstream negative-conformance program. Initial and final ASan/UBSan/LeakSanitizer checks were CLEAN for both original and candidate.

What This Demonstrates

The experiment demonstrated operational feasibility: the full pipeline from deterministic analysis through model invocation, strict validation, and independent native measurement completed without manual intervention. It did not demonstrate optimization efficiency. A technically valid candidate that passes all checks is not valuable unless it produces a proven improvement.

Salsa20 — End-to-End Pipeline Exercise

Phase 2.2 · Salsa20 from LLVM Test Suite (revision 624469628023)

What Happened

The model independently proposed interleaving four disjoint quarter-rounds at each rotation stage of the Salsa20 cipher. The candidate passed strict validation including scope/replay checks, reference matching against Bernstein's published test vectors and 264 independent expectations, and CLEAN ASan/UBSan/LeakSanitizer results for both variants.

However, the candidate measured slower than the original: 0.944× with a confidence interval entirely below 1.0, indicating a detected local regression. The pipeline correctly classified this as NO_SUPPORTED_GAIN. No retry, repair, or retiming was performed.

Why This Matters

This demonstrates the complete pipeline working correctly, including correctly rejecting an unsupported optimization. A system that only reports wins would be unreliable. The ability to detect and refuse a plausible-sounding but actually harmful change is a necessary engineering property.

Earlier Experiment Phases

The autonomous optimization infrastructure was developed iteratively. Earlier phases exercised individual gates before the full pipeline was connected.

Limitations and Conclusions

Observed Limitations

• High inference and context consumption per candidate (52K+ input tokens for a single proposal)
• Substantial validation infrastructure cost beyond the LLM call itself (compilation, sanitizer builds, measurement rounds)
• No guaranteed relationship between token consumption and achieved speedup
• A small code edit that passes all checks is not valuable unless it produces a proven runtime improvement
• No demonstrated favorable cost-benefit ratio in any measured case to date
• Model familiarity with public source code cannot be excluded; controlled novelty is not established

Conclusion

The agent integration was technically functional: the pipeline correctly invoked a model, received structured proposals, validated candidates through multiple independent gates, measured native performance, and accepted or rejected results based on evidence. But its measured optimization benefit did not justify broad adoption as the primary optimization engine.

The experiment establishes that the infrastructure works end-to-end. It does not establish that LLM-generated optimizations are generally effective, cost-efficient, or superior to deterministic analysis alone.

Provenance