Scaling down for efficiency: Medium-sized protein language models perform well at transfer learning on realistic datasets
2024

Medium-sized protein language models are effective for transfer learning

publication Evidence: moderate

Author Information

Author(s): Vieira Luiz C., Handojo Morgan L., Wilke Claus O.

Primary Institution: The University of Texas at Austin

Hypothesis

Do medium-sized protein language models perform comparably to larger models in transfer learning tasks?

Conclusion

Medium-sized models like ESM C 600M provide a good balance between performance and efficiency in transfer learning applications.

Supporting Evidence

  • Medium-sized models performed comparably to larger models when data was limited.
  • Mean embeddings consistently outperformed other compression methods.
  • Larger models require more data to maximize their utility.

Takeaway

This study shows that smaller protein models can do just as well as bigger ones when there's not much data, making them easier to use.

Methodology

The study evaluated various protein language models across multiple biological datasets to assess the impact of model size on transfer learning.

Limitations

The study's findings may not apply to all datasets, particularly those with very large protein sequences or specific types of data.

Digital Object Identifier (DOI)

10.1101/2024.11.22.624936

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication