Language Models Can Rewrite Their Own Weights ~ Ibrahim Kaya
×
Artificial Intelligence

Language Models Can Rewrite Their Own Weights

İbrahim Kaya Published: Updated: 2 min read
Language Models Can Rewrite Their Own Weights
Contents

Standard deep learning setups train a model once and leave the weights frozen. Improving reasoning afterward usually requires fine-tuning on new data or spending compute on longer reasoning chains during inference. A paper on arXiv tests an alternative route: self-rewriting, in which a model modifies its own internal weights through forward-pass reasoning, without calculating gradients or running backpropagation.

The model as its own optimizer

In this setup, the model runs the optimization steps itself. Using an open-weights 27B parameter architecture, researchers gave the system access to its own weight matrices, its past task outputs, and forward-pass execution. The model reviewed where its reasoning failed on test tasks and identified which parameters to adjust.

The experiments showed several clear outcomes:

  • Accuracy rose by 6.6 percentage points on evaluation sets the model had not seen during training.
  • The model tested candidate weight edits on scratchpad problems first, keeping only changes that improved scores.
  • Accuracy stayed steady or increased across eight consecutive rewriting passes without catastrophic forgetting.
Conceptual Forward-Pass Optimizer
# Self-rewriting loop executed purely via forward reasoning
weights = model.inspect_weights()
diagnostic_log = model.analyze_errors(inputs, expected_structure)
candidate_weights = model.synthesize_edits(weights, diagnostic_log)

# Acceptance gate without gradient backpropagation
if model.benchmark(candidate_weights) > model.baseline:
    model.commit_weights(candidate_weights)

Autonomous repair and model scale

Researchers also evaluated how well the process handled damaged weights. After corrupting parameters until the model failed routine tasks, they let it inspect the modified layers. Without access to the original checkpoint and without retraining, the model adjusted its damaged matrices and recovered over 80% of its original accuracy on visual and reasoning benchmarks.

This recovery behavior proved to be an emergent property of scale. Smaller variants in the same model family failed at the task, either producing no measurable change or breaking the layers further. Only the 27B model had enough capacity to plan and apply coherent weight edits.

Connecting training and inference

Inference-time search methods incur repeated compute costs on every query. Self-rewriting writes changes directly to the weights, so the updates remain available on future passes. Because the loop uses regular forward passes, it runs within standard inference memory budgets.

The authors note that the boundary between training and inference reflects existing software tooling rather than an absolute limit of neural networks. The approach offers a working demonstration of recursive self-improvement, showing that models can adjust their own parameters without external optimizer pipelines.