Reverse Quantization in Language Model Pretraining

A GPT-2-sized study of whether a BF16 finishing phase can recover validation quality after simulated NVFP4 training. On four H100 GPUs, a single-seed experiment found partial recovery: switching to BF16 after 80% of training recovered about 55% of the measured loss gap, but remained behind the full-BF16 baseline. NVFP4 is simulated here, and the result still needs multi-seed replication.