Algorithmic opacity

algorithmic secrecythe black box society
the idea

The unreadability of automated decision systems treated as a matter of power rather than a puzzle about how they work. The question shifts from whether a system can be understood to who is permitted to understand it — and the recurring image is the one-way mirror, where the operators see everything about a person while revealing nothing of how they act on it.

The inscrutability of automated decision systems treated as a relation of power rather than only an epistemic puzzle — opacity maintained by parties who observe everything about the subject while disclosing nothing of how they act on it. The framing is Frank Pasquale's The Black Box Society (2015).

Algorithmic opacity is the black box treated as a question of power. Pasquale keeps the cybernetic sense — inputs and outputs visible, the conversion hidden — but shifts the emphasis from whether a system can be understood to who is permitted to understand it, across reputation scoring, search ranking, and finance. His image is the one-way mirror: corporate actors with unprecedented knowledge of daily life, the subjects of that knowledge with almost none in return.

Where the machine-learning sources ask after interpretability and explainability as technical properties, opacity in this sense is a question of law and disclosure, and its concrete remedy in Wachter, Mittelstadt and Russell is the counterfactual explanation — telling a person what to change without revealing, or requiring, the model's internals.

Discussed in§

you are here in the territory

3 direct·4 two hops·16 further·46 off-graphopen the full territory →