Data: Time Series, Features & Cleaning

4.1 Real sensor data is dirty

Paper 1's plant produces thousands of measurements per day: five weather stations logging GHI (global horizontal irradiance, W/m²), ambient temperature, and — at station WS5 only — air pressure, wind direction, wind speed, and humidity; plus power from every one of the 84 inverters and the plant total. At that scale, sensor failure, communication dropouts, and faulty readings are not risks — they are certainties. The paper treats missing and faulty entries as a fact of life to be engineered around, not an inconvenience to be ignored.

Paper 1 organizes the journey from sensor to model as a three-stage pipeline:

Stage 1 just records everything. Stage 2 (Section 4.2) repairs and removes. Stage 3 (Sections 4.3–4.4) reshapes clean data into features a model can learn from.

4.2 Cleaning: eliminate, interpolate, impute

Stage 2 has exactly three operations, and you should be able to name and distinguish all three:

• Elimination — delete what is clearly wrong. Paper 1's examples: GHI readings of 1600 W/m² (physically implausible), temperatures of 50 °C, and single sensors deviating wildly from the norm of the other stations. It also eliminates curtailment periods — stretches where the plant's output was purposefully limited (e.g. on grid-operator instruction). Curtailed output is real data but not natural plant behaviour; leaving it in would teach the model that sunny skies sometimes produce capped power for no visible reason.
• Interpolation — fill short gaps, at most 1 hour, by drawing smooth values between the known endpoints. The 1-hour cap is deliberate: interpolating longer gaps would alter the character of the signal, inventing smooth weather where a storm might have passed.
• Imputation — repair long gaps, from hours up to weeks, using other sources of truth: the multi-station weather network (if WS2 died, its neighbours kept measuring the same sky), and — for missing inverter power — substituting data from inverters with similar output dynamics.

4.3 Feature engineering: giving the model what it can't compute

Stage 3 begins with a beautiful idea. The sun's position in the sky is deterministic astronomy: for any future instant, the solar Altitude β (height above the horizon) and Azimuth Φ (compass bearing) can be computed exactly — Paper 1 uses the pvlib-solarposition library. So the model is handed, as input, the exact sun position for every forecast moment:

This is future information used legitimately: unlike future cloud cover, which is unknowable without a weather service, future sun angles are pure calculation. A pleasant side effect: once the model knows exactly where the sun will be, a separate hour-of-day feature becomes unnecessary — the angles already encode the time.

Two more Stage-3 moves you met in Module 1 now earn their place in the pipeline:

• One-hot encoding of wind direction (4 entries: N/E/S/W) and month (12 entries). Cyclical variables lie when written as numbers: 359° and 0° are the same wind, December and January are neighbouring months, but numerically they sit at opposite ends of the scale. One-hot removes the false ordering.
• Normalization of all inputs (with forecasts de-normalized back to megawatts afterwards) — chiefly for computational speed: features on wildly different scales make gradient descent slow and lopsided.

4.4 Resolution and sliding windows

The plant logs at 1-minute intervals, but Paper 1 averages this down to 5-minute resolution. Counter-intuitive but wise: minute-level wiggles add unnecessary uncertainty — noise without repeatable pattern — while 5-minute data keeps every dynamic that matters for 15-minute-step forecasts.

The model's input is then a sliding window of recent history: the last 1, 2, 3, 6, or 24 hours of feature vectors, tried as hyperparameter options. Plus one clever extra:

Window length was searched per model, and the winners are telling: FFNN 1 h, LSTM 3 h, GRU 6 h. The recurrent models prospered with longer histories — they digest a sequence step by step and can prioritise within it without being overwhelmed by input size, whereas the FFNN, which swallows the whole window as one flat vector, did best keeping that vector small. This observation is exam gold; Module 5 builds on it.

4.5 Feature selection by experiment, not correlation

Which features actually help? The standard textbook answer is to compute Pearson or Spearman correlation between each candidate feature and the target, and keep the strong ones. Paper 1 explicitly declines, for a sharp reason: pairwise correlation cannot reveal feature-combination effects. A feature that looks weak alone may be valuable alongside another; a feature that correlates well may add nothing a better feature hasn't already supplied.

Instead, features were added progressively, in combinations, during the grid search — measure the model with feature set A, then A+B, then A+B+C — letting forecast accuracy itself be the judge. The verdict:

4.6 Vision data: from video to training pairs

Paper 2 faces the same "raw world → model input" problem in a different medium. Its pipeline:

• Motor babbling, disciplined. The robot performs random trajectories, but the action space is discretized at 9° intervals, and a feedback loop with the joint encoders guarantees data quality: a frame is saved only when every joint is within 0.5° of its commanded target, and a movement that fails to progress within 1 second is flagged as halting (e.g. a collision) and handled. Each saved frame is thus a verified (image, joint-angle) pair — the labels come free from the robot's own encoders.
• 12,000 data points: 10,000 for training/validation (split 8:2) and 2,000 held out for testing. Each image is RGB, resized to 100×100.
• Colour-based segmentation to binary silhouettes. The arm is painted black and blue against a clean, uniform background. The background is isolated by taking median pixel values across many images — since the robot is the only moving element, each pixel's median over time is simply the background. Then RGB thresholds (0.15 for the body, 0.4 blue for the end effector) produce binary images, with pixel values normalized to [0, 1].

Why is a flat binary silhouette enough to supervise a 3D self-model? Because the model sees thousands of silhouettes across varied poses. Any single 2D outline is ambiguous, but only the correct 3D shape is consistent with all of them at once — the same principle by which many 2D X-ray angles determine a 3D CT volume. (Module 6 makes this precise.)

4.7 Class imbalance in pixels

One last trap. In a 100×100 silhouette, the robot arm occupies a small fraction of the frame — the overwhelming majority of pixels are black (empty). A model minimizing average pixel error can therefore score deceptively well with a degenerate strategy: predict every pixel black. Paper 2's model initially fell into exactly this local optimum, producing all-black images.

Two mitigations, both worth naming precisely:

1. Early abort and restart: if the model is predicting only black images early in training, abandon that run and restart (new random initialization) rather than waste epochs in a dead end.
2. Central focus: for the first 200 iterations, concentrate the loss on the central 50×50 region — where the robot actually is — so the arm's pixels, not the empty border, dominate the early gradient signal.