MacBehaviour: An R package for behavioural experimentation on large language models
2024

MacBehaviour: An R package for behavioral experimentation on large language models

Sample size: 3 publication 10 minutes Evidence: high

Author Information

Author(s): Xufeng Duan, Shixuan Li, Zhenguang Cai

Primary Institution: The Chinese University of Hong Kong

Hypothesis

Can large language models exhibit human-like tendencies in inferring gender from novel personal names based on their phonology?

Conclusion

The study found that large language models consistently infer gender from novel names, demonstrating human-like tendencies.

Supporting Evidence

  • Large language models like GPT-3.5 Turbo and Llama-2-7b-chat-hf were tested for gender inference.
  • Results showed a significant effect of name phonology on gender attribution.
  • Models demonstrated consistent trends in gender inference across different experimental designs.

Takeaway

This study shows that computer programs called large language models can guess if a name sounds like a boy's or a girl's name, just like people do.

Methodology

The study involved three experiments using the MacBehaviour R package to test large language models on their ability to infer gender from names based on phonology.

Potential Biases

Potential biases in model responses based on training data and the specific prompts used.

Limitations

The study may not fully capture the models' capabilities since pronouns do not always have to be at the beginning of a sentence.

Statistical Information

P-Value

p<0.001

Statistical Significance

p<0.001

Digital Object Identifier (DOI)

10.3758/s13428-024-02524-y

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication