Performance Experiments and Reproducible Evidence

Measured results from optimization experiments. Each result includes its complete experimental conditions, statistical evidence, and limitations.

Independent Optimization Experiments

Six programs from the LLVM Test Suite were selected under a frozen protocol. Each candidate was independently investigated, implemented, correctness-checked, and measured. No candidate or measurement was added after final timing.

Program Candidates Correct Best Speedup Verdict
minisat 2 2 — Inconclusive
viterbi 2 1 — Inconclusive
ReedSolomon 3 3 1.103× [0.984, 1.130] Inconclusive
fftbench 1 1 1.468× [1.338, 1.547] Verified gain
himenobmtxpa 1 1 — Inconclusive
BilateralFilter 1 1 1.740× [1.590, 1.794] Verified gain

"Correct" means finite reference and differential checks pass under the registered protocol; it is not a proof over all possible inputs. Intervals are local unadjusted 95% bootstrap CIs. An interval containing 1 establishes neither equivalence nor regression freedom.

fftbench — Verified Local Gain

Program fftbench (LLVM Test Suite, degree 131,072)
Compiler Clang/LLVM 21, -O3 -g -march=znver2 -fno-omit-frame-pointer, no fast-math
CPU AMD Ryzen 5 7520U (Zen 2)
Optimization Forward FFT stage twiddle table with block-major contiguous butterflies
Measurements 16 balanced AB/BA rounds, 3 warmups per variant, pinned to CPU 0
Speedup 1.468× [1.338, 1.547]
Original / Candidate 61.87 ms / 43.26 ms median wall time
Memory Cost ~1 MiB additional transient allocation per forward transform
Correctness Reference + differential checks pass; real/imaginary forward/inverse values compared as exact IEEE bits. Matching upstream leaks retained — not sanitizer-clean
Status Verified local gain

BilateralFilter — Verified Local Gain

Program BilateralFilter (LLVM Test Suite, 100 iterations per case)
Compiler Clang/LLVM 21, -O3 -g -march=znver2 -fno-omit-frame-pointer, no fast-math
CPU AMD Ryzen 5 7520U (Zen 2)
Optimization Cached exact spatial Gaussian weights once per invocation for positive diameters ≤128; original path retained as fallback
Measurements 16 balanced AB/BA rounds, 3 warmups per variant, pinned to CPU 0
Speedup 1.740× [1.590, 1.794]
Original / Candidate 222.74 ms / 140.59 ms median wall time
Memory Cost 128 KiB bounded stack reservation
Correctness All reference + differential checks pass; complete outputs compared on mixed/constant/signed images, odd/even diameters, boundary diameters 127–129. ASan/UBSan/LSan CLEAN
Status Verified local gain

BilateralFilter — Per-Case Results

Case Paired Median Speedup Local 95% CI Outcome
16/21.515×[1.403, 1.897]gain
16/41.631×[1.530, 1.798]gain
32/21.501×[1.441, 1.615]gain
32/41.704×[1.610, 1.860]gain
64/21.641×[1.495, 1.781]gain
64/41.828×[1.650, 1.882]gain

Printed native CPU case measurements supplement the primary external whole-executable wall times. Their precision and unadjusted intervals limit inference; all six cases are retained and no measured case regression was detected.

Context and Limitations

These results come from a bounded iterative profile-guided optimization campaign. Machine-code evidence from Ghidra/LLVM informed the investigations, but this experiment has no LLVM-only control, so unique decompiler advantage is not established. Static instruction counts alone never determine a runtime verdict.

Candidates were investigated using genuine 100 Hz gprof CPU-PC profiles selecting original hotspots. Clean native baselines and final builds are separate artifacts. Instrumented self-fractions are approximate, not native Amdahl bounds; executable-text profiles miss shared-library and kernel work.

Reproducing and Evaluating a Published Optimization Example

An independent study of the MATMUL demonstration published in the Codee Open Catalog. This evaluation examines whether the Precedes deterministic engine can detect and reproduce the published loop-interchange optimization.

Variant Source Detected by Precedes Generated by Precedes Median (s) Speedup
Original Upstream intact Blocked No 8.407 1.000×
Manual Reference REFERENCE_MANUAL_LOOP_INTERCHANGE N/A No 2.244 3.746×
After resolving the AST context issue (Clang 21 parse with C17 support), the product generated a deterministic optimization:
Precedes Generated PCPP-MATMUL-INTERCHANGE-001 (AST rule + alias guard) Detected Yes 2.022 3.899×

Experimental Conditions

Precedes Version 0.3.2, commit c9a70c8
CPU AMD Ryzen 5 7520U
Compiler GCC 15.2.0, -O3 -fopenmp -std=c17, no fast-math
Protocol n=1500, 7 paired rounds, 3 warmups per variant, CPU 0, OMP_NUM_THREADS=1, OMP_DYNAMIC=FALSE
Correctness 2,590 elements compared; max original/reference difference: 0
Why BLOCKED Clang AST channel blocked by missing omp.h context (INCLUDE_ROOT_OUTSIDE_ALLOWED_TREE). The four GCC PCPP-VEC-001 warnings are generic vectorization diagnostics, not loop-interchange detection
Provider Calls 0 — no AI model was used

Deterministic Patch Result

After fixing the AST context issue (Clang 21 with C17), Precedes identified finding PCPP-MATMUL-INTERCHANGE-001 using AST-based ForStmt pattern matching. The CLI generated a loop-interchange patch with an alias guard that verifies row-disjointness at runtime using uintptr_t interval checks. When the guard fails, the original function is called.

Speedup 3.899× (ratio of medians); paired median 3.784×, range [3.594, 4.452]
Correctness 2,590 elements in 12 cases (rectangular, dyadic/decimal, p=0); max difference: 0. ASan/UBSan/LSan: CLEAN
Commit 72406bdb5befa33198b82c483cbdb9990fad9ec3
Tests 120 focused tests pass; 0 provider calls

Interpretation

The initial MATMUL study was BLOCKED due to a missing OpenMP header context. After resolving the AST path, the Precedes deterministic engine successfully detected the loop-interchange opportunity and generated a correct patch with a 3.899× measured speedup.

The manual reference (3.746×) and the Precedes-generated variant (3.899×) apply the same fundamental transformation (i-k-j reordering). The Precedes variant additionally includes a runtime alias guard. These are not directly comparable measurements — they were taken in separate acquisition campaigns.

Codee's published 3.30× corresponds to different hardware and compiler. Codee was not installed or executed. This study does not constitute a head-to-head product comparison.

Methodology Notes

Measurement Protocol

  • Temperature, CPU frequency, and outside system load were uncontrolled; they limit generalization of all results
  • Bootstrap intervals use 10,000 median resamples with a fixed seed, without multiplicity adjustment
  • Candidate selection uses preliminary feedback from 6 serial rounds; final samples (16 rounds) are independent in time
  • Intervals containing 1 establish neither equivalence nor regression freedom
  • No incompatible experimental campaigns are aggregated into a single success rate
  • Every round retains raw output, reference outcomes, wall/CPU/system time, peak RSS, round order, and binary identity hashes
  • Final data are retained once; no result was retimed to improve a number

Correctness Verification

  • Upstream reference contracts use original fpcmp comparator/tolerances and exit-status convention
  • Supplemental probes execute actual original/candidate implementations, not rewritten arithmetic models
  • Final original/candidate whole workloads and numerical drivers are compiled with ASan/UBSan including LeakSanitizer
  • Matching leak-only reports meet the registered correctness rule; unresolved active UB excludes a win
  • Finite tests and registered reasoning do not prove behavior over all possible inputs or floating-point environments

What These Results Do Not Establish

  • A general population success rate — program selection was non-random and budgets varied
  • That iteration itself caused the gains — both FFT and BilateralFilter succeeded on the first implemented candidate
  • Memory-regression freedom — additional allocations are documented per experiment
  • Universal semantic equivalence — correctness checks are finite
  • Unique decompiler advantage — no LLVM-only control experiment exists