Scaling down for efficiency: Medium-sized protein language models perform well at transfer learning on realistic datasets
2024
Medium-sized protein models are efficient for learning
publication
Evidence: moderate
Author Information
Author(s): Vieira Luiz C., Handojo Morgan C., Wilke Claus O
Primary Institution: Cold Spring Harbor Laboratory
Hypothesis
Do medium-sized protein language models perform well in transfer learning on realistic datasets?
Conclusion
Medium-sized protein language models can perform nearly as well as larger models while being more efficient.
Supporting Evidence
- Larger models do not necessarily outperform smaller ones when data is limited.
- Medium-sized models like ESM-2 650M and ESM C 600M showed good performance.
- Mean embeddings outperformed other compression methods in transfer learning.
Takeaway
This study shows that smaller protein models can be just as good as bigger ones for learning from data, which makes them easier to use.
Methodology
The performance of various protein language models was evaluated across multiple biological datasets.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website