Catastrophic forgetting

During sequential training, gradients from a new task can overwrite parameters that matter for earlier tasks. Methods such as EWC protect some of those parameters, but there is still no general solution for large language models learning from an open-ended stream of experience.