Skip to content

Latest commit

 

History

History
1 lines (1 loc) · 1.24 KB

File metadata and controls

1 lines (1 loc) · 1.24 KB
  • MLX SFT now honours training.max_grad_norm (#749 by @Srinivasan8888 in #750). The field is forwarded into TrainingArguments by sixteen transformers trainers, so the documented default clips every transformers run at 1.0 — and on backend: mlx it reached nothing: no MLX file read it, mlx_lm's TrainingArgs has no such field and its trainer clips nowhere, and the MLX backend ignores: line did not mention it. The same config therefore trained clipped on one backend and unclipped on the other, silently. Gradients are now clipped through mlx.optimizers.clip_grad_norm by an optimizer that clips before delegating, which reaches mlx-lm's single optimizer.update(model, grad) call site without forking its training loop; optimizer.state and optimizer.learning_rate still delegate, because upstream reads both off the object it is handed. The norm that ran is recorded in adapter_config.json. Measured on an M1 inside upstream's own compiled-step shape, one step on an ill-conditioned batch moved SGD weights by 388.03 unclipped versus 0.0085 at max_grad_norm: 1.0; AdamW moves 0.126505 versus 0.126504, because Adam normalises by its second moment — so this rescues optimizer: sgd / lion and gradient spikes rather than every MLX run.