Evaluation Report: iter1_baseline_gpt_700
Analysis Notes
ITERATION 1: Baseline GPT on 700-canvas dataset
================================================
Model: GPT AutoregressiveViT (default params)
- embed_dim=256, depth=8, num_heads=8, patch_size=16
- 6.9M parameters
- 100 epochs, batch_size=8, lr=1.5e-4
Dataset: 700 canvases (single-action-shoulder-pan-700-combined)
- 630 train / 70 val
- Action distribution: 345 move+ (action=1), 355 move- (action=2), 0 stay (action=0)
- Canvas: 464x480 (448 visual + 16 motor strip, 6 joints)
Purpose: Establish baseline with 7x more data than previous 100-sample run.
Previous run (100 samples): best_val=0.009839, SSIM=0.62, PSNR=17.6,
action_disc=0.016, motor_dir_acc=67.9%
Training result: best_val=0.005279 (46% improvement from more data)
Key questions for this evaluation:
1. Did the train/val gap close with more data?
2. Did action discrimination improve?
3. Is GPT error compounding still the dominant issue?
4. How does motor prediction quality change?
Metrics
| Metric | Value |
|---|
| val_mse | 0.012655 |
| val_mse_visual | 0.012655 |
| ssim | 0.696908 |
| psnr | 19.419347 |
| val_mse_motor_strip | 0.007126 |
| val_mse_action_1 | 0.013139 |
| val_mse_action_2 | 0.012093 |
| val_mse_static | 0.002685 |
| val_mse_dynamic | 0.025371 |
| motor_position_mae_mean | 0.876543 |
| motor_velocity_mae_mean | 0.063530 |
| motor_direction_accuracy | 0.868750 |
| motor_consistency_error | 1.169797 |
| gpt_teacher_forcing_loss | 0.005279 |
| gpt_free_running_loss | 0.012531 |
| gpt_tf_fr_gap | 0.007253 |
| action_discrimination_score | 0.029936 |
| motor_discrimination_score | 0.053826 |
| motor_position_mae_per_joint | [2.9007, 0.5437, 0.9453, 0.1488, 0.6802, 0.0406] |
| motor_position_mae_action_1 | 1.022885 |
| motor_position_mae_action_2 | 0.706469 |
Recommendations
Motor position and velocity predictions are inconsistent. Consider adding a consistency loss or simplifying the velocity encoding.
Counterfactual Action Grids
Each grid: Row 1 = GT, Row 2 = STAY (red), Row 3 = MOVE+ (green), Row 4 = MOVE- (blue)
GPT Per-Position Loss (last frame, raster order)
| Position | MSE |
|---|
| 0 | 0.004843 |
| 1 | 0.004338 |
| 2 | 0.003212 |
| 3 | 0.004779 |
| 4 | 0.002410 |
| 5 | 0.002137 |
| 6 | 0.000953 |
| 7 | 0.000897 |
| 8 | 0.000719 |
| 9 | 0.000594 |
| 10 | 0.000658 |
| 11 | 0.006855 |
| 12 | 0.007609 |
| 13 | 0.008151 |
| 14 | 0.005726 |
| 15 | 0.001310 |
| 16 | 0.001745 |
| 17 | 0.002418 |
| 18 | 0.001825 |
| 19 | 0.001108 |
| 20 | 0.000997 |
| 21 | 0.001029 |
| 22 | 0.001051 |
| 23 | 0.001899 |
| 24 | 0.006937 |
| 25 | 0.007255 |
| 26 | 0.009636 |
| 27 | 0.010152 |
| 28 | 0.002039 |
| 29 | 0.001908 |
| 30 | 0.000742 |
| 31 | 0.001558 |
| 32 | 0.001227 |
| 33 | 0.000491 |
| 34 | 0.000950 |
| 35 | 0.000657 |
| 36 | 0.001118 |
| 37 | 0.004137 |
| 38 | 0.006188 |
| 39 | 0.007774 |
| 40 | 0.007824 |
| 41 | 0.009640 |
| 42 | 0.002102 |
| 43 | 0.001634 |
| 44 | 0.001679 |
| 45 | 0.001309 |
| 46 | 0.001055 |
| 47 | 0.000649 |
| 48 | 0.000896 |
| 49 | 0.000779 |
| 50 | 0.001851 |
| 51 | 0.005636 |
| 52 | 0.006273 |
| 53 | 0.006065 |
| 54 | 0.007580 |
| 55 | 0.008797 |
| 56 | 0.000835 |
| 57 | 0.001168 |
| 58 | 0.000915 |
| 59 | 0.000603 |
| 60 | 0.000678 |
| 61 | 0.000808 |
| 62 | 0.000786 |
| 63 | 0.000726 |
| 64 | 0.001569 |
| 65 | 0.006118 |
| 66 | 0.005586 |
| 67 | 0.005688 |
| 68 | 0.005527 |
| 69 | 0.008137 |
| 70 | 0.001004 |
| 71 | 0.001027 |
| 72 | 0.000630 |
| 73 | 0.000453 |
| 74 | 0.000596 |
| 75 | 0.000386 |
| 76 | 0.000353 |
| 77 | 0.001910 |
| 78 | 0.003930 |
| 79 | 0.003209 |
| 80 | 0.001977 |
| 81 | 0.003468 |
| 82 | 0.003614 |
| 83 | 0.003228 |
| 84 | 0.001715 |
| 85 | 0.001041 |
| 86 | 0.000562 |
| 87 | 0.000339 |
| 88 | 0.000407 |
| 89 | 0.000305 |
| 90 | 0.000277 |
| 91 | 0.000528 |
| 92 | 0.001756 |
| 93 | 0.001913 |
| 94 | 0.001214 |
| 95 | 0.002339 |
| 96 | 0.003840 |
| 97 | 0.002351 |
| 98 | 0.000746 |
| 99 | 0.000722 |
| 100 | 0.000640 |
| 101 | 0.000536 |
| 102 | 0.000437 |
| 103 | 0.000330 |
| 104 | 0.000330 |
| 105 | 0.000815 |
| 106 | 0.001018 |
| 107 | 0.001284 |
| 108 | 0.000681 |
| 109 | 0.001065 |
| 110 | 0.001290 |
| 111 | 0.004436 |
| 112 | 0.001129 |
| 113 | 0.000527 |
| 114 | 0.000550 |
| 115 | 0.000576 |
| 116 | 0.000517 |
| 117 | 0.000417 |
| 118 | 0.000373 |
| 119 | 0.000781 |
| 120 | 0.000956 |
| 121 | 0.000716 |
| 122 | 0.000432 |
| 123 | 0.000400 |
| 124 | 0.001863 |
| 125 | 0.006926 |
| 126 | 0.000797 |
| 127 | 0.000440 |
| 128 | 0.000439 |
| 129 | 0.000533 |
| 130 | 0.000519 |
| 131 | 0.000299 |
| 132 | 0.000268 |
| 133 | 0.000433 |
| 134 | 0.000560 |
| 135 | 0.000725 |
| 136 | 0.000388 |
| 137 | 0.000616 |
| 138 | 0.003617 |
| 139 | 0.005174 |
| 140 | 0.000317 |
| 141 | 0.000239 |
| 142 | 0.000261 |
| 143 | 0.000267 |
| 144 | 0.000328 |
| 145 | 0.000253 |
| 146 | 0.000294 |
| 147 | 0.000409 |
| 148 | 0.000606 |
| 149 | 0.000791 |
| 150 | 0.001231 |
| 151 | 0.002570 |
| 152 | 0.003553 |
| 153 | 0.004642 |
| 154 | 0.000319 |
| 155 | 0.000258 |
| 156 | 0.000232 |
| 157 | 0.000259 |
| 158 | 0.000295 |
| 159 | 0.000230 |
| 160 | 0.000294 |
| 161 | 0.000177 |
| 162 | 0.000209 |
| 163 | 0.000632 |
| 164 | 0.000943 |
| 165 | 0.005024 |
| 166 | 0.001502 |
| 167 | 0.004027 |
| 168 | 0.000314 |
| 169 | 0.000229 |
| 170 | 0.000197 |
| 171 | 0.000235 |
| 172 | 0.000224 |
| 173 | 0.000250 |
| 174 | 0.000256 |
| 175 | 0.000235 |
| 176 | 0.000187 |
| 177 | 0.000203 |
| 178 | 0.000690 |
| 179 | 0.001500 |
| 180 | 0.000480 |
| 181 | 0.001742 |
| 182 | 0.000260 |
| 183 | 0.000231 |
| 184 | 0.000226 |
| 185 | 0.000222 |
| 186 | 0.000279 |
| 187 | 0.000236 |
| 188 | 0.000474 |
| 189 | 0.000572 |
| 190 | 0.000464 |
| 191 | 0.000323 |
| 192 | 0.000312 |
| 193 | 0.001135 |
| 194 | 0.000912 |
| 195 | 0.001413 |
| 196 | 0.025451 |
| 197 | 0.016685 |
| 198 | 0.014141 |
| 199 | 0.014177 |
| 200 | 0.012128 |
| 201 | 0.014981 |
| 202 | 0.020606 |
| 203 | 0.021069 |
| 204 | 0.022623 |
| 205 | 0.018999 |
| 206 | 0.020197 |
| 207 | 0.026655 |
| 208 | 0.018792 |
| 209 | 0.020055 |
| 210 | 0.019154 |
| 211 | 0.016210 |
| 212 | 0.010705 |
| 213 | 0.012972 |
| 214 | 0.011106 |
| 215 | 0.010687 |
| 216 | 0.016270 |
| 217 | 0.022554 |
| 218 | 0.017890 |
| 219 | 0.016418 |
| 220 | 0.016083 |
| 221 | 0.012017 |
| 222 | 0.009457 |
| 223 | 0.011365 |
| 224 | 0.019695 |
| 225 | 0.015597 |
| 226 | 0.011073 |
| 227 | 0.013625 |
| 228 | 0.010256 |
| 229 | 0.010029 |
| 230 | 0.014167 |
| 231 | 0.017066 |
| 232 | 0.016075 |
| 233 | 0.016814 |
| 234 | 0.011123 |
| 235 | 0.008899 |
| 236 | 0.009232 |
| 237 | 0.007224 |
| 238 | 0.016828 |
| 239 | 0.011391 |
| 240 | 0.013375 |
| 241 | 0.012189 |
| 242 | 0.008872 |
| 243 | 0.007739 |
| 244 | 0.011257 |
| 245 | 0.015136 |
| 246 | 0.013977 |
| 247 | 0.016142 |
| 248 | 0.012630 |
| 249 | 0.008530 |
| 250 | 0.009430 |
| 251 | 0.008909 |
| 252 | 0.013592 |
| 253 | 0.009558 |
| 254 | 0.010632 |
| 255 | 0.010821 |
| 256 | 0.009237 |
| 257 | 0.007196 |
| 258 | 0.009152 |
| 259 | 0.013031 |
| 260 | 0.013779 |
| 261 | 0.012466 |
| 262 | 0.010057 |
| 263 | 0.007795 |
| 264 | 0.006594 |
| 265 | 0.006512 |
| 266 | 0.011282 |
| 267 | 0.006524 |
| 268 | 0.008110 |
| 269 | 0.007580 |
| 270 | 0.008749 |
| 271 | 0.007899 |
| 272 | 0.009374 |
| 273 | 0.010576 |
| 274 | 0.013120 |
| 275 | 0.009921 |
| 276 | 0.005853 |
| 277 | 0.005572 |
| 278 | 0.005271 |
| 279 | 0.004949 |
| 280 | 0.010664 |
| 281 | 0.007397 |
| 282 | 0.008621 |
| 283 | 0.007710 |
| 284 | 0.008719 |
| 285 | 0.009898 |
| 286 | 0.009119 |
| 287 | 0.013306 |
| 288 | 0.009166 |
| 289 | 0.008753 |
| 290 | 0.005926 |
| 291 | 0.004311 |
| 292 | 0.006350 |
| 293 | 0.005394 |
| 294 | 0.009174 |
| 295 | 0.010061 |
| 296 | 0.007325 |
| 297 | 0.007421 |
| 298 | 0.010495 |
| 299 | 0.005913 |
| 300 | 0.009006 |
| 301 | 0.014707 |
| 302 | 0.009792 |
| 303 | 0.007263 |
| 304 | 0.003950 |
| 305 | 0.003340 |
| 306 | 0.005117 |
| 307 | 0.002248 |
| 308 | 0.006894 |
| 309 | 0.004815 |
| 310 | 0.005312 |
| 311 | 0.006394 |
| 312 | 0.002016 |
| 313 | 0.009896 |
| 314 | 0.008098 |
| 315 | 0.007185 |
| 316 | 0.008793 |
| 317 | 0.005231 |
| 318 | 0.004513 |
| 319 | 0.003758 |
| 320 | 0.002188 |
| 321 | 0.002118 |
| 322 | 0.007122 |
| 323 | 0.004108 |
| 324 | 0.009072 |
| 325 | 0.002131 |
| 326 | 0.000647 |
| 327 | 0.012720 |
| 328 | 0.010072 |
| 329 | 0.001848 |
| 330 | 0.008278 |
| 331 | 0.006240 |
| 332 | 0.003238 |
| 333 | 0.002761 |
| 334 | 0.001436 |
| 335 | 0.001589 |
| 336 | 0.006540 |
| 337 | 0.006195 |
| 338 | 0.006620 |
| 339 | 0.000657 |
| 340 | 0.000578 |
| 341 | 0.008179 |
| 342 | 0.009520 |
| 343 | 0.001505 |
| 344 | 0.004120 |
| 345 | 0.006665 |
| 346 | 0.003401 |
| 347 | 0.002281 |
| 348 | 0.001475 |
| 349 | 0.002099 |
| 350 | 0.013373 |
| 351 | 0.011191 |
| 352 | 0.005178 |
| 353 | 0.000552 |
| 354 | 0.000488 |
| 355 | 0.016753 |
| 356 | 0.009345 |
| 357 | 0.002157 |
| 358 | 0.000982 |
| 359 | 0.005085 |
| 360 | 0.004556 |
| 361 | 0.001536 |
| 362 | 0.001750 |
| 363 | 0.001077 |
| 364 | 0.009603 |
| 365 | 0.011765 |
| 366 | 0.002339 |
| 367 | 0.000391 |
| 368 | 0.000649 |
| 369 | 0.011375 |
| 370 | 0.006675 |
| 371 | 0.002528 |
| 372 | 0.000681 |
| 373 | 0.000947 |
| 374 | 0.003385 |
| 375 | 0.003098 |
| 376 | 0.001542 |
| 377 | 0.000945 |
| 378 | 0.013377 |
| 379 | 0.006084 |
| 380 | 0.000320 |
| 381 | 0.000313 |
| 382 | 0.001006 |
| 383 | 0.009562 |
| 384 | 0.008211 |
| 385 | 0.002551 |
| 386 | 0.000370 |
| 387 | 0.000253 |
| 388 | 0.001020 |
| 389 | 0.002406 |
| 390 | 0.001378 |
| 391 | 0.001035 |
| 392 | 0.002001 |
| 393 | 0.002197 |
| 394 | 0.001101 |
| 395 | 0.000345 |
| 396 | 0.020397 |
| 397 | 0.000099 |
| 398 | 0.000220 |
| 399 | 0.023983 |
| 400 | 0.021714 |
| 401 | 0.000239 |
| 402 | 0.022358 |
| 403 | 0.002617 |
| 404 | 0.000057 |
| 405 | 0.000054 |
