Pokemon TCG training status — rendered 2026-07-17 07:20 UTC
541,640
trainer step
4,403 steps/h
541
checkpoints
541 published
361,480
replay files
≈games
250
worker jobs
1.238
eval policy
2.568
eval draft
uniform = 7.14
1.555
eval value
random ≈ 1.0
series: policy ·
draft · value
Training loss (by step, log y)Holdout eval loss (by step)Trainer progress (steps over hours)Replay window (samples over hours)Learning rate (by step, log y)
Knobs — fleet config, applied to new workers on save
loading…
change log
Deployment probe (argmax, no search) vs training agent
09:25 replay window 500k -> 10k files; window_files added as a dynamic knob (trainer re-reads knobs.json each refresh; <=0 = unbounded)
09:54 pacing v3 after restart stall (lifetime samples_seen had drifted to 2.1M vs ~28M real): generated-samples counted from replay dir per-file, ckpt high-water floor
06:27 forced-aggression ON at step 325k (ckpt_00325000): force_aggression_p=0.5, force_aggression_c=0.5, turn_count_penalty=0.002; trainer kept running through the flip; fleet converges via container turnover
21:55 forced-energy E range knobs deployed (~step 393k): force_energy_min/max (E ~ uniform inclusive) via knobs.json + driver pass-through + dashboard panel; set 0/30 matching the prior hardcode; worker image rebuilt, driver restarted
23:08 400k pause executed by armed stop_at_400k.sh: ckpt_00400000 published; tg-train, tg-driver, and all tgjob workers killed; inference servers + probe keeper left running
23:37 resume after 400k pause: driver fleet relaunched (250 workers, same params/knobs: energy p=1 E~U(0,30), aggression 0.5/0.5, turn penalty 0.002); trainer container had survived the stop (docker outlives tmux) ~40 steps past 400k, stalled on the pacing gate, unblocked on fresh samples; train.log pipe reattached (tg-trainlog)
07:26 full-distribution forced targets ON: trainer relaunched with --draft-full-softmax --draft-weight 0.2 (draft CE over full 1267-card vocab, zero-target ids now pushed down; draft_loss rescaled, non-comparable with history); resumed mid-generation (400k gen cosine)
07:29 worker fleet cutover to full-candidate recording: masked (forced-aggression) decisions now store the full pre-mask candidate list with visit mass on attacks and zero on non-attacks - direct attack-vs-pass policy targets; trainer needs no change (candidate softmax unchanged)
21:20 pipelines recadenced: probe keeper -> 25k-step grid from 500k (lowest unprobed first); auto-submitter -> 50k step multiple (~2.2 submissions/day of the 5/day cap)
21:54 500k anneal executed by armed anneal_at_500k.sh: knobs -> energy p=0.5 E~U(5,15), aggression p=0.25 c=1.0, window_files 50000; trainer relaunched --gen-steps 50000 (LR sawtooth now 50k-period), generation start at 500000; fleet converged via turnover (flags verified)