Learning More May Not Be Better: Knowledge Transferability in Vision-and-Language Tasks
Author Information
Author(s): Chen Tianwei, Garcia Noa, Otani Mayu, Chu Chenhui, Nakashima Yuta, Nagahara Hajime
Primary Institution: Institute for Datability Science, Osaka University
Hypothesis
Is learning more knowledge always better for vision-and-language models?
Conclusion
The study shows that not all knowledge transfers well or positively impacts related tasks, even when they share a common goal.
Supporting Evidence
- Tasks in the same group are more likely to help each other improve performance.
- Negative results show that tasks with shared goals do not always contribute positively to one another.
- Different random seeds strongly affect the numeric performance of each task.
- The dataset size is an important but not always positive factor in knowledge transferability.
Takeaway
Sometimes, learning more doesn't help; it can even make things worse for models that understand both pictures and words.
Methodology
The study involved hundreds of cross-experiments on twelve vision-and-language tasks categorized into four groups.
Potential Biases
The study acknowledges potential biases in large models that may encode harmful representations.
Limitations
The study primarily focuses on one-to-one transfer learning and does not explore more complex transfer scenarios.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website