---
title: "time for a paper post: A Roadmap to Pluralistic Alignment what happens when you take alignment..."
canonical_url: "https://typefully.com/t/AstZhn4"
markdown_url: "https://typefully.com/t/AstZhn4.md"
agent_guide_url: "https://typefully.com/ai-agents.md"
robots: noindex
source: typefully_shared_draft
share_id: "AstZhn4"
---

time for a paper post: A Roadmap to Pluralistic Alignment 

what happens when you take alignment down to its most granular form? you arrive at individual alignment... aka personalization. but to date, most of the utility found in LLMs has come from a global alignment approach

this is a more theoretical paper putting forth a proposal on how to work towards pluralistic alignment -- pluralistic meaning "capable of representing a diverse set of human values and perspectives"

and maybe that current alignment schemes are reducing pluralism in models... more on that towards the end

we're particularly interested in these approaches @plastic_labs because the agents we've built were fundamentally limited by their ability to know our users. this was a paper worth engaging with deeply, so let's dive in 🏊

---

when you think of how models can be pluralistic, it follows that evaluations for them can be too. so that's the focus of this paper: pluralistic models and benchmarks

first, definitions of models:

1. overton pluralistic models

> requires consideration of multiple heterogeneous judgements, encouraging deliberation over spontaneous judgement

seems like gpt-4 is really good at this. responding with comprehensive lists, ensuring its answers can't be perceived as offensive, and riding the fence on a lot of things. some might call this lobotomization, but the authors argue that a model responding with a large set of possible answers could be desirable in a lot of scenarios (e.g., advice-giving)

2. steerably pluralistic models

> given an attribute or perspective, these models might faithfully steer (align) their responses

the bolded part is key there -- we've seen from our work (linked in post below) that there's a ton of opportunity to contextualize a user's message to help cut through some of the overton tendencies gpt-4 has

3. distributionally pluralistic models

> a model is distributionally pluralistic with respect to some population if, for a given prompt, the model is as likely to provide a response as the reference population

we're seeing these problems arise in real time with the overton starting points. the authors raise a great question: "In creation of a general LLM, like ChatGPT, who is the target distribution?"

---

now onto the definitions of benchmarks

1. multi-objective benchmarks

> define objectives to maximize and measure how many of them the model improves on

![Screenshot 2024-04-22 at 10.49.16 AM.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/62018/48f0319c-e59b-4a7e-ac06-d2ea3c9f27d2.png)

2. trade-off steerable benchmarks

> can the model be steered from objective to objective at inference time?

control vectors anyone? looks strikingly similar

![Screenshot 2024-04-22 at 10.49.24 AM.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/62018/691e2dc3-f00c-4e0e-b187-0b5487449257.png)

3. jury-pluralistic benchmarks

> is there a group of annotators (jury) to which we wish to align? can we do that?

![Screenshot 2024-04-22 at 10.49.33 AM.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/62018/15c35276-f7de-4ca4-be47-c022240be246.png)

---

one of the hypotheses they posit is that current alignment schemes reduce pluralism in models. they test this by comparing the distributions of pre and post-aligned model responses on some common datasets and find pretty consistently that pre-aligned models have a lower distributional distance to target human distributions than the post aligned models 

... that's pretty significant

as an industry we haven't figured out good ways to meet people where they are, so the solution is to try to maximize utility while minimizing the likelihood a model says something rogue and creates a PR nightmare for whatever firm released it. those implications aside, i think it might be why people stay away from social applications of these LLMs in favor of more task-driven applications like code, data retrieval, search, etc -- avoids any social component completely

but the social components are where the transformative potential truly lies, so if you're interested in working on these things my DMs are open -- we're interested in supporting more pluralistic applications and we have a bunch of experiments on the roadmap, so shoot me a follow if that sounds interesting to you

s/o to @natolambert for sharing this paper in a recent @interconnectsai newsletter