Temporal Simultaneity

Temporal Simultaneity Predicts Annotation Quality in Setswana Sentiment Corpora

Researchers present a Setswana sentiment dataset of 3,565 tweets annotated by three native speakers across eight batches, finding that inter-annotator agreement (IAA) declines sharply over time despite an aggregate Kappa of 0.76. The dominant predictor of agreement quality is temporal simultaneity: tweets labeled within one minute achieve κ=0.98 versus κ=0.65 for those labeled more than a day apart. The study also benchmarks multilingual encoders and proprietary models including GPT-5 and Gemini on three-class sentiment classification, with GPT-5 few-shot achieving the best result at 62.2 macro-F1. The dataset, timestamps, and analysis code are released to support reproducible quality auditing for African language NLP.