MacBehaviour: An R package for behavioral experimentation on large language models
Author Information
Author(s): Xufeng Duan, Shixuan Li, Zhenguang Cai
Primary Institution: The Chinese University of Hong Kong
Hypothesis
Can large language models exhibit human-like tendencies in inferring gender from novel personal names based on their phonology?
Conclusion
The study found that large language models consistently infer gender from novel names, demonstrating human-like tendencies.
Supporting Evidence
- Large language models like GPT-3.5 Turbo and Llama-2-7b-chat-hf were tested for gender inference.
- Results showed a significant effect of name phonology on gender attribution.
- Models demonstrated consistent trends in gender inference across different experimental designs.
Takeaway
This study shows that computer programs called large language models can guess if a name sounds like a boy's or a girl's name, just like people do.
Methodology
The study involved three experiments using the MacBehaviour R package to test large language models on their ability to infer gender from names based on phonology.
Potential Biases
Potential biases in model responses based on training data and the specific prompts used.
Limitations
The study may not fully capture the models' capabilities since pronouns do not always have to be at the beginning of a sentence.
Statistical Information
P-Value
p<0.001
Statistical Significance
p<0.001
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website