Self-Supervision & Implicit 3D Models

6.1 Learning without labels

Supervised learning needs labelled examples, and labels usually mean expensive human annotation. Self-supervised learning sidesteps this: the supervision signal is generated from the data itself. Paper 2's robot is its own annotator. During motor babbling — random exploratory movement, like an infant flailing to discover its limbs — every frame automatically pairs two things the robot already has:

• its joint angles, read directly from its own motor encoders, and
• what the camera saw at that instant.

That pairing is the labelled dataset: 12,000 frames, no human annotation anywhere. The model's job is to predict the image from the angles; reality grades the prediction.

6.2 Implicit representations: models you query

Explicit 3D representations store the shape directly: meshes (triangles), CAD models, point clouds. An implicit representation stores no geometry at all — instead, a neural network is the shape:

Ask the function about any point in space and it answers. Three properties make this powerful:

• Continuous: query anywhere, at any resolution — no fixed grid.
• Compact: Paper 2's whole self-model fits in 333 kB, versus 1.1 MB for previous approaches.
• Differentiable: gradients flow through it — the property Section 6.6 cashes in.

6.3 NeRF in one page

Neural radiance fields (NeRF) learn a 3D scene from many ordinary 2D photos. The recipe:

• A network maps (3D position, viewing direction) → color + density — an implicit model of the scene.
• Volume rendering: to predict one pixel, march along that pixel's camera ray, sampling many points; accumulate density×color along the ray into a single predicted pixel value.
• Training: compare predicted pixels against the real photos and backpropagate the error.

The remarkable part: no 3D supervision anywhere. Because every photo constrains what the field must look like from that viewpoint, many 2D images jointly pin down the 3D structure. The network is never told the geometry — it is forced to invent geometry that explains all the pictures at once.

6.4 The FFKSM: NeRF's robot cousin

Paper 2's free-form kinematic self-model keeps NeRF's core bet — an implicit field trained only against 2D images — but changes three things:

• Silhouettes, not RGB: the ground truth is a binary image (robot / not-robot), so no colors are modelled.
• A kinematic input: the network also receives joint angles, making it a model of an articulated moving body, not a frozen scene. Change the angles and the predicted shape changes with them.
• Density σ and visibility α as outputs — not NeRF's density-plus-transmittance rendering.

The full pipeline, one query point at a time:

Three fully-connected networks, three jobs. The coordinates encoder C(·) sees the query point after it has been transformed into virtual coordinates by the first two joints' rotations, X′ = T⁻¹X with R_yaw(A₀) and R_pitch(A₁) — geometrically equivalent to the camera moving while the robot base stays put, so those two joints never need to be learned. The kinematic encoder K(·) handles the remaining angles A₂, A₃. The predictive module P(·) fuses both encodings and outputs, for query point k on the camera ray through pixel (i, j):

Density σ says "is my body here?"; visibility α says "does this point actually influence what the camera sees?". Their product, summed along the ray, is the predicted pixel; the MSE against the 100×100 binary camera image is the entire loss. Everything the model knows was forced into it by that one comparison.

6.5 Positional encoding

Why not feed raw (x, y, z) straight into the coordinates encoder? Because neural networks are biased toward smooth functions of their inputs — raw low-dimensional coordinates make it very hard to learn sharp spatial detail like a thin robot link against empty space. The fix, standard across the NeRF family, is to expand each input through sines and cosines at multiple frequencies — 5 frequencies here — before the network sees it:

• coordinates: 3 numbers → 33 dimensions,
• joint angles: 2 numbers → 22 dimensions.

The high-frequency features give the network ready-made "wiggly" basis functions, so representing fine structure no longer fights its smoothness bias. This is the mirror image of one-hot encoding from Module 1.7: there a raw number said too much (false order); here a raw number says too little (no fine-scale handles).

6.6 What a differentiable self-model buys you

Because the FFKSM is differentiable end-to-end, gradient descent can optimize not just the weights during training but the inputs after training. That single fact powers all three of Paper 2's demonstrations:

• Gradient-based inverse kinematics: freeze the weights, treat the joint angles as the variables, and nudge them (Adam, lr 0.04) so the predicted end-effector position moves toward a target. No hand-derived kinematic equations — the learned model replaces them.
• Collision checking: query the density field along a candidate motion; non-zero σ where an obstacle sits means a collision. A queryable body model turns planning into asking questions.
• Self-monitoring: keep comparing camera reality against model prediction. When they diverge, something about the body changed — damage — and the same self-supervised loop that trained the model can fine-tune it back to accuracy on fresh post-damage data.