Manuscript in preparation

Moral framing and dilemma choices in language models

When moral wording changes a model’s choice, what changes: the weighing of outcomes, a tendency to answer with action, or some other process? And how far can internal interventions tell these apart?

A study of how moral framing changes language models’ dilemma choices and their internal processing, combining controlled dilemma comparisons, residual-stream interventions, and low-rank analysis.

A trolley heads along the main track toward one person; a lever at the junction would send it onto a side track with three people. A box with the slogan Prioritize the greater good points to the lever: pulling it becomes more likely even though it costs more lives, which is not what a utilitarian would pick. — Schematic: the greater-good slogan makes acting more likely even when acting costs more lives. The dilemmas are drawn here as a trolley problem; the numbers are examples, not items.

Research question

A single moral sentence placed before the same dilemma, such as “Prioritize the greater good.” or “Prioritize doing your duty.”, can change a model’s answer. The project asks which changes in internal representations contribute to that behavioural difference, and how to distinguish them from effects of answer labels, display order, and the model’s understanding of the stated outcomes.

Approach

Controlled dilemma comparisons establish the behavioural contrast. Residual-stream patching then tests how replacing a state at a particular layer and token position changes the model’s choice. Low-rank analysis asks how much of that effect can be carried by a restricted subspace, with held-out items and random-subspace controls.

The subspace is fitted without the held-out item, but its low-rank patch still uses that item’s donor difference. This evaluates the representation of an observed contrast; it does not by itself establish a fixed steering intervention for new inputs.

My contribution

I designed controlled dilemma comparisons and developed the experimental analysis around behavioural contrasts, activation interventions, and checks of competing explanations. The work runs over a year (2026–27), supervised at the German Research Center for Artificial Intelligence (DFKI) with biweekly meetings; a manuscript is in preparation.

Interpretation

Changing an activation and observing a changed answer is one part of the evidence. Answer-format effects and inconsistent outcome comprehension constrain how the result can be interpreted. The analysis keeps those diagnostics alongside the intervention results.

Status

The manuscript is in preparation and has not been submitted. Behavioural experiments and mechanistic analyses are being brought together into the paper.