Tina Shahedi

About Me

I'm Tina Shahedi, a Junior Researcher with hands-on experience in AI and NLP projects at Utrecht University. My work has focused on large-scale data applications, and I'm passionate about using technology to solve real-world problems.

I'm now eager to take the next step in my academic journey and am actively seeking PhD opportunities where I can apply my background in Industrial Engineering, data science, and NLP to meaningful research. I'm especially interested in contributing to projects that combine innovation with social impact. You can learn more about my work on my personal website and view my CV here.

One of my most recent research papers in NLP was accepted at the GeBNLP Workshop at ACL 2025:

Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation

ACL 2025
Hadi Mohammad1, Tina Shahedi1, Pablo Mosteiro1, Massimo Poesio2, Ayoub Bagheri1, Anastasia Giachanou1
1 Department of Methodology and Statistics, Utrecht University
2 Queen Mary University of London, London, United Kingdom
Paper arXiv

Figure 1. We instruct LLMs to replicate human annotations for subjective NLP tasks from different perspectives using persona prompting and XAI techniques.

Abstract

Understanding the sources of variability in annotations is crucial for developing fair NLP systems, especially for tasks like sexism detection where demographic bias is a concern. This study investigates the extent to which annotator demographic features influence labeling decisions compared to text content. Using a Generalized Linear Mixed Model, we quantify this influence, finding that while statistically present, demographic factors account for a minor fraction (~8%) of the observed variance, with tweet content being the dominant factor. We then assess the reliability of Generative AI (GenAI) models as annotators, specifically evaluating if guiding them with demographic personas improves alignment with human judgments. Our results indicate that simplistic persona prompting often fails to enhance, and sometimes degrades, performance compared to baseline models. Furthermore, explainable AI (XAI) techniques reveal that model predictions rely heavily on content-specific tokens related to sexism, rather than correlates of demographic characteristics. We argue that focusing on content-driven explanations and robust annotation protocols offers a more reliable path towards fairness than potentially persona simulation.

Key Findings

  • - Gender and age group do not significantly influence labeling decisions.
  • - Black annotators are far more likely to label tweets as sexist, and Latino annotators are less likely to do so compared to White annotators.
  • - Annotators with a high school degree are significantly less likely to label tweets as sexist.
  • - Annotators from Africa are significantly less likely to label tweets as sexist.

GenAI Results

  • - Baseline GenAI models show strong performance on this task.
  • - Adding demographic persona information provides inconsistent benefits for improving annotation reliability.
  • - Guiding model attention using XAI based on content features shows more consistent improvements. This approach is especially effective when combined with capable models.
  • - Focusing on the text itself through XAI methods appears more promising than relying on potentially superficial persona simulation.

Mixed-effect Model

We ran a mixed-effects logistic regression model to understand how annotators’ demographic features affect their labeling decisions.

BERT Model

  • We fine-tuned a multilingual BERT model for sexism detection, using class weights to handle label imbalances and early stopping to prevent overfitting.
  • We evaluate the reliability of GenAI models as annotators, including the effect of persona prompting and XAI-guided explanations.
  • We analyze the impact of demographic and content features on model predictions and annotation reliability.

Example results or figures from the study (update or replace as needed).

Paper

BibTeX

@inproceedings{mohammadi2025assessing,
  title={Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation},
  author={Mohammadi, Hadi and Shahedi, Tina and Mosteiro, Pablo and Poesio, Massimo and Bagheri, Ayoub and Giachanou, Anastasia},
  booktitle={The 6th Workshop on Gender Bias in Natural Language Processing},
  pages={92},
  year={2025}
}