Scaling down for efficiency: Medium-sized protein language models perform well at transfer learning on realistic datasets
2024

Medium-sized protein models are efficient for learning

publication Evidence: moderate

Author Information

Author(s): Vieira Luiz C., Handojo Morgan C., Wilke Claus O

Primary Institution: Cold Spring Harbor Laboratory

Hypothesis

Do medium-sized protein language models perform well in transfer learning on realistic datasets?

Conclusion

Medium-sized protein language models can perform nearly as well as larger models while being more efficient.

Supporting Evidence

  • Larger models do not necessarily outperform smaller ones when data is limited.
  • Medium-sized models like ESM-2 650M and ESM C 600M showed good performance.
  • Mean embeddings outperformed other compression methods in transfer learning.

Takeaway

This study shows that smaller protein models can be just as good as bigger ones for learning from data, which makes them easier to use.

Methodology

The performance of various protein language models was evaluated across multiple biological datasets.

Digital Object Identifier (DOI)

10.1101/2024.11.22.624936

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication