Evaluation, Uncertainty & Fair Comparison

7.1 Deterministic metrics recap — and their politics

Quick recap from Module 2, one line each (all normalised by the plant's rated capacity P_cap where relevant):

• NRMSE: root-mean-square error, normalised — squares the errors, so large errors are punished disproportionately.
• MAE: mean absolute error — directly interpretable in physical units (Paper 1's best models sit around 750 kW).
• MAPE: absolute error as a percentage of capacity — comparable across plants and horizons.

The politics: which metric you optimize or select on is a choice with consequences. Paper 1 selects its models on NRMSE precisely because large forecast errors carry real cost — grid stress, financial loss — while reporting MAE for operator interpretability. A model can rank differently under different metrics: model A can beat model B on MAE yet lose on NRMSE if A makes occasional huge mistakes. That freedom to pick the flattering metric is factor (1) of the superiority critique in 7.6.

7.2 Point forecasts are not enough: probabilistic evaluation

A forecast that says "the plant will produce 42.3 MW at 14:00" hides everything that matters about risk. Is that 42.3 ± 1 MW or 42.3 ± 15 MW? A grid operator scheduling reserves needs the band of plausible outcomes, not a single number. Confidence intervals supply exactly that: a quantified statement of how certain the forecast is.

Paper 1 names the standard alternatives — quantile regression, kernel density estimation — and picks bootstrapping for its simplicity and proven track record in forecasting research.

7.3 The bootstrap, step by step

The bootstrap is non-parametric: it assumes no distribution for the errors — no "suppose errors are Gaussian". The data speaks for itself. The recipe:

1. Take your N test errors.
2. Sample N of them with replacement (some drawn twice, some not at all) — one bootstrap sample.
3. Compute the statistic of interest on that sample — here the mean (Paper 1 uses MAPE as the statistic).
4. Repeat m = 10,000 times, giving 10,000 bootstrap estimates of the mean.
5. The spread of those 10,000 means gives the 95% confidence interval.

7.4 Slice your results honestly

An aggregate metric averages over regimes and hides where the model actually struggles. Paper 1 therefore reports performance per weather type and per forecast step:

The interesting engineering lives in the hard slices: intermittent days — clouds racing across the field — are precisely where inverter-level models actually helped, a finding a single aggregate number would have buried. Paper 2 slices the same way: whole-body vs end-effector-only accuracy, and robot 1 vs robot 2 (different morphology) vs the damaged robot 3. Whenever you present results, an examiner may ask: where does it fail? Sliced reporting is the honest answer format.

7.5 Baselines: the art of fair comparison

A result means nothing without a reference point. Paper 2's morphology-prediction baselines are chosen thoughtfully:

• Random selection (RS): answer any query by picking a random training image. A distribution-matched lower bar — what you get for free.
• Nearest neighbour (NN): pick the training image whose joint angles are closest (L2 distance) to the queried ones. A strong non-learning baseline: similar joint angles produce similar images, so NN is genuinely hard to beat with only 12,000 training frames to memorize.

Whole-body morphology MSE (in 2D image px²): theirs (OM) 0.004 vs NN 0.010 vs RS 0.029 — the learned model beats the honest NN baseline by 2.5× (and end-effector-only: 0.001 vs 0.003 vs 0.004). Beating RS alone would have proven almost nothing; beating NN shows the model generalizes beyond retrieving memorized frames.

7.6 The “claimed superiority” critique

Paper 1's Section 4.1 is unusually blunt: almost every forecasting paper claims its model is superior, yet the claims are founded on different scenarios and largely cannot be compared or trusted. Four factors:

1. Different performance metrics utilised. Each paper picks its own metric (7.1) — rankings don't transfer.
2. Benchmarking against less capable models. Beating a weak or outdated baseline proves little (7.5).
3. Incomplete test data sets. Short or cherry-picked test periods — excluding hard weather types or seasons — are not representative of true generality. Test set length is a primary indicator of generality; Paper 1 tests on a full unseen year, all seasons and weather types.
4. Biased model optimisation effort. The proposed model gets weeks of careful tuning; the baselines get default settings. The comparison measures effort, not architecture.

The remedy: transparency plus equal, systematic optimisation effort for every model compared — which is what Paper 1's 3-phase hyperparameter framework delivers (Module 8).

7.7 Evaluation limits: know what your metric cannot see

Paper 2 admits a real limitation: its self-model is only ever evaluated through 2D image error. Direct ground-truth measurement of the robot's 3D morphology is unattainable without extra equipment (3D scanners, motion capture) — the very equipment the method exists to avoid. So the paper's 3D claims rest on 2D projections: a shape could in principle be wrong in depth while projecting correctly, and the metric would never notice. The authors acknowledge this openly.

The generalizable lesson: every metric has a blind spot — know yours and say so. NRMSE can't see whether errors cluster in one weather regime; MAPE hides absolute magnitude; 2D silhouette MSE can't see depth. Naming your evaluation's limits, as both papers do, is a mark of scientific maturity examiners reward.