‎

Take a prediction market specified by distributions \(P(X_i\mid Y)\) and \(P(Y)\) with all variables binary.

Suppose Trader 1 moves the probabilities to \(P(Y=1)=\frac12\) and \(P(X_i=1\mid Y=y)=y\)

And Trader 2 moves \(P(Y=1)\) to \(0.9\).

And say the markets resolve to \(X_i=1\) for all \(i\). This outcome had probability \(1/2\) according to Trader 1 (whereas maybe before the trader it had probability \(1/2^N\)). If the trader

According to Trader 2 it had probability 0.9.

But what if someone flips \(P(Y=1\)) to 0.1 while also flipping \(P(X_i=1\mid Y=y)=1-y\).

Based on the log score, the agent is still correctly rewarded.

But what if the trader instead got stocks of \(Y\) — with the idea that

Suppose (letting all variables be binary) you have a bunch of ground-truth-resolvable markets \((X_1,\dots X_{10})\) in this framework and introduce a latent \(Y\) that induces conditional independence between all of them. Maybe the distributions are initialized as \(P(Y)=1/2\) and \(P(X_i\mid Y)=1/2\), so the initial probability of any particular observed outcome is \(1/2^{10}\).

A trader moves these probabilities to \(P(Y)=0.9\) and \(P(X_i\mid Y)=\{0.9\text{ if } Y;0.5\text{ else}\}\). The market resolves to (or probabilities move really close to) \(X_i=1\forall i\). Clearly this trader has added value to the market and we would like to reward him \(\log(0.9^{11}+0.5^{10}\cdot0.1)-\log(1/2^{10})\). But

Main questions:

Owning stocks of \(Y\) won’t work, so the exact algorithm from Laskey et al cannot be applied directly.
How to incentivize traders to actually bet on the latents rather than directly on the observed variables? Maybe for our purposes that’s fine because LLMs bet on everything anyway — but we’ll maybe need an LLM to “summarize/interpret” each latent based on its relation to other variables.
How to manage subsidies for latents?

Update on the latent variables stuff—

The ideal way to organize forecasts on latent variables looks something like this:

Questions are organized into a Directed Acyclic Graph, including both “observable” questions (things that will resolve) and “latent” questions (things that only obtain their meaning from their correlations with other questions)
Traders can make forecasts on any of the probabilities \(P(X\mid \mathrm{parents}(X))\) — and can also make other forecasts, which are converted into such probabilities via a maximum-entropy method (“[[https://en.wikipedia.org/wiki/Radical_probabilism#Conditioning_on_an_uncertainty_%E2%80%93_probability_kinematics][]]”).
Traders can make any edit to the DAG structure: add a new observable or latent variable; add an arrow between any two variables as long as it does not introduce a cycle; remove a sufficiently weak arrow or latent. Weak arrows and latents should also be pruned automatically, but it is important to let traders do this too so they can add new arrows without creating cycles. They can also submit multiple operations at once.
When observable questions resolve, their ground truth is backpropagated to directly compute scores for each bet that had influenced that question’s probability.

The most similar implementation to this is the SciCast team’s “graphical market-maker” — however, I don’t believe this allows including latent variable questions.

The key difference in their approach is in the last point — instead of backpropagating ground truth to directly compute scores, they backpropagate ground truth to update probabilities for each variable — and then traders can sell or whatever.

The problem with this approach for latents is that latent variables have certain fundamental “symmetries”: e.g. a latent \(Y\) can represent either “Russia wins the war” or “Russia loses the war” as long as you also similarly change all the conditional probabilities \(P(X_i\mid Y)\) to use the new latent. So a whale trader can come and totally change the “meaning” of \(Y\) to make those who had bet \(Y\) up lose money, even if they had improved the forecast.

(actually you don’t even need a whale trader — even slight reparameterizations of \(Y\) would work.)

So we need to implement a generalization of the scicast market that works with latent (or generally “ghost”) variables — I expect this might be a non-trivial effort, because in particular: it becomes non-obvious how to incentivize traders to bet on latents rather than directly on the variables themselves, keeping knowledge about the correlations private to themselves within their trading strategy. For example if a trader believes some variables are correlated through a latent, it could either exploit this by creating this latent and betting on it, or by adjusting its trading strategy to e.g. bet up \(P(X_2)\) when \(P(X_1)\) increases etc. based on its private estimates of \(P(X_2\mid X_1)\) and \(P(X_2\mid\lnot X_1)\).

The general way to get around this is by subsidizing the latent variables and their edges directly; but (1) this creates a free-money glitch and (2) it’s not clear how to, given the scoring is based on the subsidy on the observable markets.

Another thing is I’m also not totally sure that “scoring based on effect on observable variables” makes total sense; like: does it correctly reward an agent for betting up a probability on an edge that is later learned to be important? And how can we distinguish this from the possibility that the meaning of the edge has just changed?

I can work on this, but it will be a pretty big undertaking (like a full research paper worth of effort) — do you want me to prioritize this now?