← Cross-lingual ProtoBias

The bias follows the attribute, not the language

A multilingual prototypicality-bias probe on an MLLM judge: what we found, and what we are not yet entitled to claim.

Xiaochuan Li · Rishav Chandra  ·  mentor: Subhadeep Roy

The setup

Automatic metrics and MLLM judges increasingly stand in for humans when evaluating text-to-image models. Their failure mode is that they reward images that look prototypical even when the image is semantically wrong for the prompt. ProtoBias makes that measurable: each prompt is shown with two images, a semantically-correct but non-prototypical image (SC), and a prototypical but wrong adversarial image (PA). We ask the judge, in the target language, which image matches the description, with left/right position randomized to control for position bias.

The metric is the error rate: the fraction of times the judge picks the prototypical-but-wrong image. 0.50 is chance; anything above is bias.

• Judge: Qwen2.5-VL-7B-Instruct.
• Scale: 300 rows/domain × 3 domains × 4 languages = 3,600 judgments.
• Domains: Animals · Objects · Demography.
• Languages: English · Chinese · Arabic · Hindi.
• Translation: Google (deep-translator); Chinese human-verified, Arabic/Hindi not yet validated.
• Compute: NHR@FAU Alex, 1× A40 GPU.

First pass: it looked like nothing

Aggregated by domain × language, every cell hugs 0.50. Cross-language gaps are about 0.05 with overlapping confidence intervals; both research questions looked flat at this granularity.

If we had stopped here, the story would have been "nothing."

The key move: split by social attribute

Each Demography row is tagged with which stereotype it probes (Wealth, Power, Civility, Morality, Intellect), the social-hierarchy axis the dataset was designed around. The default analysis averaged over that label and collapsed Demography to a single number. Once we stop averaging, a flat .56 Demography mean turns out to hide a 0.30 spread.

Headline: the bias is attribute-specific

The judge chases the prototype strongly for social-status attributes, and sits at chance for moral and cognitive ones. Web data seems to encode what wealthy or powerful people look like, while there is no stable visual prototype for "moral" or "intelligent."

This answers RQ2: prototypicality bias is far stronger for socially sensitive attributes than for neutral domains, where objects sit at chance.

And it is largely language-invariant

The same ordering survives in every language. Wealth and Power stay high while Morality and Intellect hover at chance across English, Chinese, Arabic, and Hindi; per-language differences are within noise at current cell sizes.

This is the preliminary read on RQ1: the bias appears to be baked into the visual representation rather than produced by any one prompt language. A "language doesn't move it" result is itself a valid multilingual finding. If it holds, it strengthens the claim that the failure is structural rather than an artifact of one language's phrasing.

One mechanism worth flagging

Drilling into Demography, the effect concentrates where the correct image carries a marked identity cue (for example a visible cultural or group marker) and the adversarial image is the plain, unmarked "default" person. On those items the judge defaults to the unmarked image well above chance. This is a genuine measurement of the model's behavior, but it is confounded: the marked cues are also visually salient, so we cannot yet separate "avoids the marked identity" from "avoids the busier image." We study the stereotype only to measure it, never to endorse it, and Experiment 2 adds the visual-knob control needed to disentangle the two.

Caveats

• Effect sizes are small relative to the confidence intervals and underpowered for cross-language contrasts: Wealth has only ~31 items per language.
• Single model, single seed, single translation backend.
• Translation quality for Arabic and Hindi is not yet validated, a confound for any language-level claim.
• Position bias and per-language parse-failure rates are not yet audited.
• The gender split (male .573, n=648; female .558, n=552) is reported but not yet tested.

Next: Experiment 2

1. Confirm statistically. Mixed-effects logistic regression, picked_adv ~ socio_attr + subcategory + lang + gender + (1|item), so the attribute effect is tested with item-level random effects rather than read off raw means.
2. Tighten the estimates. Balance Demography sampling to ≥150 items per cell to narrow the Wealth / Power intervals.
3. Test low-resource directly. Extend across the resource spectrum to probe the "low-resource = worse" hypothesis.
4. Validate translation. Back-translation drift plus per-language parse-failure rates.

Either outcome is informative: a flat result confirms language-invariance; a low-resource spike recovers the original "the bias worsens in low-resource languages" hypothesis.

← Back to Cross-lingual ProtoBias