Coalitional agency thoughts

Take two agents A and B which action sets \(X_A\) and \(X_B\) and utility functions \(U_A(x_A, x_B)\), \(U_B(x_A, x_B)\). In general if they are individually maximizing their utility functions, then maybe their chosen actions \((x_A^*,x_B^*)\) will be some Nash-equilibrium of the game — but it may not be possible to interpret this as the action of a “super-agent”.

There are two ways to interpret the words “the action of a super-agent”:

• as the action that maximizes some “total utility function”, where this total utility function has some sensible properties (like being increasing in each agent’s utility)
• if the two agents could co-ordinate and choose their action, there is no other action that would be better for both of them — i.e. is Pareto-optimal. In fact it is a basic result in welfare economics that these two definitions are (under some assumptions) equivalent. And furthermore by the fundamental theorems trade achieves Pareto-optimality — i.e. trade allows the collective to be taken as a super-agent.

More precisely by “trade” we mean: perfect information, zero transaction costs, and (I guess the last one is only relevant in economics) competitive market structure. I guess it also depends on perfect property rights/perfect assurance, which is implicitly assumed under zero transaction costs.

We discussed this a bit over our call, but: “trade aligns incentives” requires two basic assumptions: (a) that there is no information asymmetry (b) there is perfect assurance, equivalently property rights/contract enforcement.

(In general I prefer this formulation to “outer and inner alignment” — the former is outer alignment; the relationship of the latter to inner alignment is more subtle but the idea is that there would be no “inner alignment problem” if your reward mechanism reliably persists forever; but once the AI gets smart enough it can just completely upend your reward mechanism.)

For instance the second fundamental theorem of welfare economics states a correspondence between “initial endowments” (which is a special case of “property rights allocations”) and “aggregate utility functions”.

I don’t know if right now I have a more precise idea than “property rights are probably worth studying; e.g. we should at least think about how to economically model settings where agents can violate rights/the initial rights allocations aren’t stable”.

I wrote some stuff about this as an undergrad: https://arxiv.org/pdf/2107.09651 — I don’t know if it’s very useful, but I did discover that there is interesting mathematical structure there, which I generally take as a sign of usefulness.

And as always, another question is how “property rights” translates into a thermodynamic context.

i.e. on the attached paper. The key questions to work on are: - how can we make market-based agents more efficient — either by introducing “parent programs” or figuring out how to backpropagate into the space of agents - how can we get “modularity of state” in an arbitrary RL setting? - proving that market-based agents maximize e.g. infinite-horizon average reward in the long run