Understanding and Mitigating the Forgetting Curve in LLMs
September 2, 2026
The forgetting curve, often referred to as catastrophic forgetting in the context of Large Language Models (LLMs), describes the phenomenon where a model loses previously acquired knowledge or capabilities when fine-tuned on new tasks. This occurs because fine-tuning alters shared internal representations, and updates for a new task can interfere with parameters supporting older tasks, leading to a sharp decline in performance on those older tasks. Even with minimal-parameter adaptation techniques like LoRA, forgetting can still occur as shared directions are moved.
The Forgetting Curve in LLM Fine-Tuning
The concept of the forgetting curve, originally studied in psychology by Hermann Ebbinghaus, describes the rate at which memories are lost over time. In LLMs, this manifests as "catastrophic forgetting" during fine-tuning. When an LLM is fine-tuned on a new task, the optimization process pushes model parameters to reduce loss on this new task. However, these same parameters often support older tasks, and without proper constraints, performance on those older tasks can drop sharply, even if the new task's loss improves.
Why Catastrophic Forgetting Occurs
Catastrophic forgetting arises because fine-tuning changes shared internal representations within the model. Even with parameter-efficient fine-tuning (PEFT) methods like LoRA, which reduce the number of parameters moved, a shared set of directions is still updated. If these directions encode capabilities needed for earlier tasks, the adapter can shift them to fit the new task distribution, causing the composed model (base + adapter) to lose older competence.
A key risk in using PEFT is mis-specifying the subspace. If the adapter family (e.g., LoRA rank or placement) or soft-prompt capacity cannot express the necessary update directions, the model can underfit. Conversely, using too large an update magnitude, such as a high learning rate or too many unfrozen parameters, can force movement along directions that harm base capabilities, leading to forgetting.
Optimization and Regularization's Role
Optimization and regularization techniques play a crucial role in shaping which directions an LLM learns and, consequently, in preventing or exacerbating the forgetting curve. Fine-tuning often behaves like "directional filtering," where a large learning rate or long training horizon allows the optimizer to move extensively along gradient-aligned directions, potentially leading to overfitting and forgetting.
Regularization methods and optimizer hyperparameters determine whether gradient updates remain "on-manifold" for the target task or drift into directions that fit training data but harm generalization. For instance, weight decay (L2) and early stopping can prevent movement in directions that do not consistently reduce validation loss.
Mitigating the Forgetting Curve
Mitigating the forgetting curve involves adding constraints, capacity allocation rules, or "memory" of past data to prevent new updates from erasing previously learned knowledge. The goal is to bias the optimization step to preserve performance on a reference set, such as past data, behaviors, or preserved subspace components.
Strategies for Forgetting Mitigation
Several strategies can be deployed to combat catastrophic forgetting:
- Rehearsal (Buffer Past Examples): This involves mixing a small, curated sample from previous tasks with new-task data. This ensures that the adapter's update still "sees" earlier behaviors, directly reducing regressions.
- Regularization (Penalize Drifting): A loss term can be added to discourage large changes in the adapter directions or their induced outputs relative to the previous model. This keeps training focused on necessary changes.
- Smaller Adaptation Subspaces: When working with small datasets, preferring smaller adaptation subspaces can implicitly regularize the model by preventing large parameter movements.
- Reduced Update Magnitude: If forgetting of base capabilities is observed, reducing the update magnitude (e.g., lower learning rate, fewer steps) or constraining parameters (e.g., PEFT, freezing more layers) can help.
- Monitoring Validation Curves: Ignoring validation curves is a common mistake. Training loss can drop while generalization fails, indicating that spurious directions were learned or sharp minima were overfit. Monitoring these curves helps identify when the optimizer is moving in harmful directions.
Monitoring and Evaluation for Forgetting
Effective monitoring and evaluation are crucial to detect and address the forgetting curve. This involves tracking various metrics beyond just training loss.
| Metric Type | Specific Metric | Purpose |
|---|---|---|
| Quality | Held-out data | Matches product goal |
| Efficiency | Trainable params | Compares across runs |
| Robustness | Hard examples | Prevents overfitting |
| Generalization | Perplexity | Catches damage |
It is important to track a quality metric on held-out evaluation data that aligns with the product goal, as perplexity alone might improve without delivering desired behavior. Efficiency metrics like trainable parameter count, training FLOPs/GPU-hours, and inference latency/cost should also be tracked to compare across runs, as PEFT aims to reduce memory and compute. Reporting a quality-efficiency curve, using multiple subspace sizes or ranks, can reveal intrinsic subspace minimality rather than just optimization quirks. Finally, running robustness slices (e.g., hard examples, longer contexts, different prompt formulations, domain shifts) helps ensure that constrained updates do not overfit the "easy" regions of the data. A baseline likelihood/generalization signal, such as perplexity, can also catch cases where constrained adaptation damages broader language behavior.
Frequently Asked Questions
What is the Ebbinghaus forgetting curve in psychology?
The Ebbinghaus forgetting curve, from psychology, illustrates the rate at which memories are lost over time if there is no attempt to retain them. In LLMs, this concept is analogous to catastrophic forgetting, where models lose previously learned information.
How does the curve of forgetting apply to LLMs?
In LLMs, the curve of forgetting, or catastrophic forgetting, describes the phenomenon where fine-tuning on new tasks causes the model to lose competence on older tasks. This happens because shared internal representations are altered during the optimization process for the new task.
What is catastrophic forgetting in the context of LLMs?
Catastrophic forgetting is the sharp decline in performance on previously learned tasks when an LLM is fine-tuned on a new task. It occurs because the model's parameters, which support both old and new tasks, are updated to minimize loss on the new task, potentially interfering with older knowledge.
Can PEFT methods like LoRA prevent the forgetting curve?
While PEFT methods like LoRA reduce the number of parameters moved, they do not entirely prevent catastrophic forgetting. LoRA updates its own trainable directions, and if these directions encode capabilities for earlier tasks, they can shift to fit the new task, leading to a loss of older competence.
What are common mistakes that lead to forgetting during fine-tuning?
Common mistakes include ignoring validation curves, using too large an update magnitude (high learning rate or too many unfrozen parameters), and using under-capacity adapters (LoRA rank or prompt length too small), which can lead to underfitting or learning spurious directions.
How can I measure the impact of the forgetting curve on my LLM?
To measure the impact, track a quality metric on held-out evaluation data relevant to your product goal, efficiency metrics like trainable parameter count, and run robustness slices on hard examples or different domains. Also, monitor a baseline generalization signal like perplexity.
Conclusion
The forgetting curve, or catastrophic forgetting, is a significant challenge in fine-tuning Large Language Models, where new learning can erase old knowledge. This phenomenon stems from the alteration of shared internal representations during optimization for new tasks. However, by implementing strategic mitigation techniques such as rehearsal, regularization, careful hyperparameter tuning, and robust evaluation practices, developers can effectively manage and reduce the impact of the forgetting curve, ensuring that LLMs retain their broad capabilities while adapting to new domains.
Sources & References
- [2504.07097] Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning
- Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation
- Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies
- Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning
- Parameter-Efficient Subspace Optimization for LLM Fine-Tuning
- Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs
- Parameter-Efficient Subspace Optimization for LLM Fine-Tuning Yuchen Lou∗
- Sculpting subspaces: How we solved continual learning in LLMs | Red Hat Developer
- Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide | Transactions of the Association for Computational Linguistics | MIT Press
- Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models | ACM Transactions on Software Engineering and Methodology
Want to actually learn forgetting curve?
Curo turns topics like this into a personalized, guided learning board - built around what you already know. Free to start.
