Training a network tunes its weights — but someone must also choose its hyperparameters: how many layers, how many units, what batch size, which inputs. Paper 1 turns that messy choice into a documented, three-phase procedure, adds a clustering trick to make 84 models affordable, and Paper 2 shows that when gradients exist, you may not need to search at all.
Hyperparameter space is effectively infinite — any number of layers, any layer width, any window length, any feature subset — while compute is finite. The usual escape routes are unstructured trial-and-error ("organic" tweaking) or pure random search. Both can find decent models, but neither can account for the effort applied: nobody can tell whether model A beat model B because it is better, or because it happened to receive more tuning.
A grid search evaluates every combination in a bounded grid of candidate values. Phase 1 first iterates over input-feature combinations and sliding-window sizes (1, 2, 3, 6, 24 h, plus the HISIMI+x historic-similarity window) — features are added progressively rather than pre-filtered by correlation analysis, because pairwise correlations cannot reveal how feature combinations behave. Then comes the hyperparameter grid proper: hidden layers (HL) × hidden units (HU) × mini-batch (mb) size.
Two deliberate simplifications tame the grid: HU sizes are powers of 2 (2x), which improves runtime on GPUs, and every hidden layer gets the same width, collapsing a combinatorial explosion of layer shapes into one dimension. The initial bounds are chosen empirically, from sampled trial runs, to likely contain the best solutions — a minimum of 240 iterations per model in Phase 1 alone.
What if the best Phase-1 solution sits on the edge of the grid? Then the true optimum may lie just outside. Phase 2 — a guided grid search, also called coordinated descent — extends the grid in exactly that direction and re-evaluates, repeating until accuracy saturates in all directions.
The paper's Fig. 11 walks through it. Suppose the best model has HU = 128 with the grid ending at 128: HU is an edge case, so extend the HU axis (add 256, 512) and retrain. The new best lands at HU = 256, HL = 3 — and HL = 3 is now the edge of the layer axis, so extend HLs to 4 (also probing one HU size smaller and larger of the other hyperparameter as you go). If the extended grids yield no further improvement in any direction, the solution space is saturated and Phase 2 is done.
Throughout Phases 1–2, mini-batch size is held at 32 and 64: empirically these worked well, because smaller batches combined with a small learning rate act as regularisers, helping solutions generalise — and changes in HU/HL move accuracy far more than changes in mb. Only in Phase 3, with the architecture settled, is the mb-size swept properly: 16, 32, 64, 128, 256. Constants throughout: Adam (lr 10−4, decay rates 0.9/0.999), ReLU activations, and early stopping with patience 20 to prevent over-fitting (drop-out was tried and helped nothing).
The inverter-level experiment needs a model for each of 84 inverters. Replicating the full-stack grid search 84 times is computationally prohibitive. The paper's solution — the inverter-clustering technique — rests on a bet: inverters with similar power dynamics will want similar hyperparameters.
Every inverter's power time series is compared to every other's (whole-clustering), giving the 84×84 matrix. K-means in two sentences: assign each point to its nearest cluster centroid, then recompute each centroid as the mean of its members; repeat until stable (here: scikit-learn, 500 iterations, 20 different centroid seeds). K-means needs K up front, so three standard guides were consulted, all pointing to K = 10:
Valid clusters must satisfy two criteria: (A) high temporal correlation (members' outputs move together in time) and (B) similar power output capacity (amplitude amplifies signal dynamics, which may demand different model complexity). Unsurprisingly, physically neighbouring inverters cluster together — they share a microclimate. Only the 10 representative inverters get the full hyperparameter search (with the top-2 combinations kept, to hedge within-cluster uncertainty); every other inverter inherits its cluster's hyperparameters but still trains its own weights.
Paper 2 closes the toolbox with a twist: sometimes you need no search at all. Its self-model is differentiable end-to-end, so controlling the robot becomes an optimization over inputs: freeze the trained model's weights, and adjust the joint angles by gradient descent to minimize the distance between the predicted end-effector position and the target — Adam again, lr 0.04, stopping at a loss below 10−5 or 1000 iterations. The same machinery that trained the network now steers the arm.
For collision-free planning, gradient descent alone is not enough (it happily walks through obstacles), so Paper 2 combines it with RRT (rapidly-exploring random tree) using its two learned models: the whole-body FFKSM as collision checker and the end-effector FFKSM as the distance heuristic guiding the tree toward the goal.