Catastrophic Forgetting

Appears in 1 paper · 2 tutorials

When a model loses knowledge from pretraining while being fine-tuned on new data.

As used in Paper 15 — Training Language Models to Follow Instructions with Human Feedback →

When a model loses knowledge from pretraining while being fine-tuned on new data. In RLHF without KL penalty, the RL policy might optimize solely for reward, forgetting useful general knowledge. The KL penalty term prevents this by keeping the policy anchored to the SFT model.

As used in Retrieval-Augmented Generation (RAG) →

When aggressive fine-tuning degrades a model's previously-good general abilities.

As used in Fine-Tuning & Model Customization →

When fine-tuning on a narrow task makes the model worse at things it used to do well. A key risk to manage. (M05)

Paper 15 — Training Language Models to Follow Instructions with Human Feedback → Retrieval-Augmented Generation (RAG) → Fine-Tuning & Model Customization →