Evaluation Report: iter6_gpt_384d12_cosine

ITERATION 6: Best architecture (384d12) with cosine LR schedule + warmup ========================================================================= Changes from Iteration 2 (best so far): - Same architecture: GPT 384d12, 22M params - LR schedule: ReduceLROnPlateau -> Cosine annealing with linear warmup - Warmup: 5 epochs of linear warmup from lr/10 to lr - Peak LR: 2e-4 (slightly higher since warmup provides stability) - Min LR: 1e-6 (cosine decays to this) - epochs: 150 (more epochs to benefit from cosine schedule) Rationale: Iterations 4-5 showed that scaling model capacity beyond 22M params provides no benefit with 700 canvases. We've hit the data ceiling. However, we haven't optimized the LR schedule. ReduceLROnPlateau is a reactive schedule — it only reduces LR after loss plateaus. Cosine annealing with warmup is the standard for ViT training and provides: 1. Stable early training (warmup prevents early divergence) 2. Better exploration of loss landscape at peak LR 3. Smooth convergence (cosine decay avoids abrupt LR drops) Architecture comparison so far: - iter1: GPT 256d8 (6.9M) -> SSIM=0.697, val_mse=0.0127 - iter2: GPT 384d12 (22M) -> SSIM=0.756, val_mse=0.0102 *** BEST *** - iter3: MAE 384d8 -> SSIM=0.616, val_mse=0.0147 - iter4: GPT 384d16 (29M) -> SSIM=0.730, val_mse=0.0133 - iter5: GPT 512d12 (38M) -> SSIM=0.746, val_mse=0.0110 Key insight: 384d12 is the sweet spot for 700 canvases. This iteration keeps that architecture and optimizes training dynamics.