Rich, Fair, or Rich and Fair: Gender Bias Mitigation in Image Captioning

Image captioners misclassify women far more than men: across UpDown, Transformer, and ClipCap, female error rates ran 2–3× higher, and stereotyped context leaked into predictions (e.g., women playing tennis captioned as men). I built a modular Bias Mitigation Module (BMM): a lightweight demographic classifier whose features are fused into a frozen captioner by weighted addition at inference, steering it toward correct gender predictions with no retraining of the base model. Three variants were tested on all three architectures over COCO val2014: Rich (9K Academic headshots, 7 attributes), Fair (87K FairFace images, 3 attributes), and a Rich+Fair curriculum hybrid. They were scored on Gender Error, BiasAmp, and LIC plus five caption-quality metrics with paired t-tests on stratified buckets. The headline finding: debiasing is architecture-dependent. The right module/architecture pairing produced statistically significant cuts in female misclassification and bias amplification while retaining 97–99% of caption quality (ClipCap + Fair BMM, for instance, dropped female error from 4.5% to 3.7%). The work also contributes the first significance-tested evaluation framework for the task, where prior work reported only aggregate deltas.

Dataset

Academic headshots (10K), FairFace (87K), COCO val2014 (evaluation)

Models

UpDown, Transformer, ClipCap, Bias Mitigation Module

Evaluation

Gender bias (Gender Error, BiasAmp, LIC), caption quality (BLEU, METEOR, ROUGE-L, CIDEr, SPICE)

Results & insight

Debiasing is architecture-dependent