Differential learning kinetics govern the transition from memorization to generalization during in-context learning
2024
How Learning Changes from Memorization to Generalization in Transformers
publication
Author Information
Author(s): Nguyen Alex, Reddy Gautam
Primary Institution: Cornell University
Hypothesis
The transition from memorization to generalization in in-context learning is governed by the relative learning rates of independent sub-circuits.
Conclusion
The study reveals that the rates at which memorization and generalization sub-circuits learn explain the transition from memorization to generalization in transformers.
Supporting Evidence
- The study identifies a memorization scaling law that determines the task diversity threshold for generalization.
- It explains various phenomena related to in-context learning, such as the long-tailed distribution of ICL acquisition.
Takeaway
This study shows that transformers learn differently when they memorize or generalize information, and understanding this can help improve how they learn.
Methodology
The study uses both theoretical analysis and experimental approaches with a small transformer on a synthetic in-context learning task.
Want to read the original?
Access the complete publication on the publisher's website