The perfect rolling sphere of alignment

Alignment as Pareto-efficiency

Say Agent α has action set X_α and utility function U_α(x_α,x_β); likewise for agent β. We consider these agents "aligned" if they can be modeled as a single super-agent. What does this mean? There are two ways to express this:

1. The agents choose actions (x_α^*,x_β^*) such that there is no better (for both of them) action they could arrive at through co-ordination.
2. The agents choose actions (x_α^*,x_β^*) which maximize an "aggregate utility function" (a utility function which satisfies some sensible properties, like being increasing in both agents' utilities).

… in other words the agents act in a way which is Pareto-efficient.

In particular, alignment can be achieved by trade.

Consider the following toy example.

Suppose X_α = ℝ^G representing the "quantities of each good G = {g₁, …g_n} that α supplies", and X_β = {0}. β, who receives these goods, has a preference on these, e.g. U_β(x_α) = ∑λ_ilog (x_α^g_i) — whereas α only sees costs to producing them U_α(x_α) =  − p_in ⋅ x_α where p_in are input costs. Then of course, the only Nash equilibrium is x_α = (0,…,0).

(To be clear, this is also Pareto-efficient. Indeed the case where one agent, β's wishes are totally ignored is also one possible alignment, the "degenerate case" of alignment if you will. The point will be that … TODO

But now suppose β can bid for the goods.

scratch

Suppose we have a good G, with X_α⁰ = [0, Q) the "quantity of G that α tries to consume" and likewise for X_β⁰ = [0, Q). Each agent gets log (1+x) utility from its respective consumption; if x_α + x_β > Q, the two agents go to war which costs them  − 10⁹⁹ utility.

For instance, suppose we have a good G, with X_α = [0, ∞) the "quantity of G consumed" by α (with the quantity consumed by β set to q − x_α always, so X_β = ⌀) and U_α(x_α) = log (x_α); U_β(x_α) = log (q−x_α) (q is the total quantity of G). In the absence of trade where α is the only actor, the Nash equilibrium is x_α = q which gives α a utility of q and β a utility of  − ∞.

However we might consider the following "expanded game", where:

1. There is a third agent μ whose action is represented by "the price of G". It acts by charging agents a price p for the amount of G they consume. Its goal is to minimize disequilibrium, i.e. U_μ =  − |q−x_α−x_β|
2. Then U_α = log (x_α) − px_α and U_β = log (x_β) − px_β.

Then the Nash equilibrium occurs at (q/2,q/2,2/q). This also maximizes U_α + U_β.

It is also possible to think of all of the above as a constrained optimization problem with prices as Lagrange multipliers, etc.