Aggregating utility across time
Standard decision theory studies one-shot decisions, where an agent faces a single choice. Real decision problems, one might think, are more complex. To find the way out of a maze, or to win a game of chess, the agent needs to make a series of choices, each dependent on the others. Dynamic decision theory (aka sequential decision theory) studies such problems.
There are two ways to model a dynamic decision problem. On one approach, the agent realizes some utility at each stage of the problem. Think of the chess example. A chess player may get a large amount of utility at the point when she wins the game, but she plausibly also prefers some plays to others, even if they both lead to victory. Perhaps she enjoys a novel situation in move 23, or having surprised her opponent in move 38. We can model this by assuming that the agent receives some utility for each stage of the game. The total utility of a play is the sum of the utilities of its stages.
This approach is popular in AI. But it has strange consequences if a decision can affect the number of future stages.
Suppose you have a choice between a short but happy life and a very long but barely tolerable life. The present approach says you should opt for the second kind of life. This seems "repugnant". It's surely not irrational to prefer the short and happy life!
The present approach also assigns a surprisingly important role to the scaling of utility. It is usually thought that utility functions are unique only up to positive linear transformation. On the present approach, however, the position of the zero makes a big difference. A long life with utilities [1,1,1,1,1,1,1,1,1,1,1] is better than a short life with utilities [10], but if we move the scale by subtracting 2, the long life [-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] comes out much worse than the short life [8].
We might try to avoid these problems by averaging rather than summing over future utilities. The average utility of a barely tolerable long life is worse than that of the short but happy life, and this doesn't change if we rescale the utilities.
But the average measure is problematic in other respects (familiar from population ethics). It implies that a long and terrible life, [-10,-10,-10,-10,-10,…], is better than a short and slightly more terrible life, [-11]. It also implies that it's better to extend a happy life [10,10,10,10,10,10] by adding a miserable stage [-5] at the end than by adding a few OK stages [3,3,3]. I suppose you might have these preferences, but surely they are not general demands of (structural) rationality!
All this suggests that we should adopt a different approach to dynamic decision problems. In philosophy and theoretical economics, it is usually assumed that a dynamic decision problem has a single "outcome" or "consequence", brought about by the entire series of choices. The agent finds the way out of the maze or not; they win the game of chess or not. This outcome has a fixed utility that doesn't depend on the path by which the outcome is reached.
This doesn't seem right either. The agent might care about the path. They might prefer some plays over others.
Let's assume, then, that the "outcome" isn't just the final stage but the final stage together with its history – i.e., the entire sequence of stages. Unlike in the previous approach, we don't assume that the utility of the sequence is separable into utilities for each stage in the sequence.
What does the utility of a sequence of stages represent? It is not some kind of aggregate of how much the agent likes each stage when it is reached. This was the previous approach, and it didn't work. No, the utility of a sequence of stages simply measures the agent's preference for that sequence over others.
But preferences can change over time. (Think of the Ulysses problem, much discussed in dynamic decision theory.) Which preference are we talking about? The preferences that are relevant at the first choice point must be the agent's preferences at that time.
(Another question in dynamic decision theory is how the agent should think about her future choices. The most popular answer is that the agent should treat her future self the way she would treat a different agent. This seems right to me. If we put these two lessons together, it turns out that all decision problems are really one-shot decision problems. The agent's later choices fall under the "states" and the "outcomes" reflect any aspect of a sequence of choices and its consequences that matters to the agent at the current stage.)
OK. Now suppose we're not interested in the theory of practical rationality, but in ethics. What should an agent do from a purely moral perspective?
A simple form of utilitarianism says that the ultimate bearers of moral value are agent stages: what ultimately matters is well-being, and well-being is a property of agents at times. A morally optimal choice is one that maximizes the total well-being of all agent stages in the world. It follows that if you have to choose between a short but happy life and a long but barely tolerable life, and your choice doesn't affect anyone else, then you are morally required to choose the long and barely tolerable life.
I find this completely incredible. It's not immoral to prefer the short and happy life!
(I've assumed the more popular "total" version of utilitarianism. The "average" version fares even worse, for the reasons mentioned above.)
To fix this problem, utilitarians should reject the assumption that moral value is separable across stages of a person's life: the ultimate bearers of value are not agent stages but entire lives. (But why stop there? Once you recognize that value isn't separable across time, why assume it is separable across agents?)
OK. Now suppose we're not interested in ethics, nor in practical rationality, but in epistemology. We want to know what an agent should believe, from a purely epistemological perspective.
It's tempting to apply the resources of decision theory to this question, as suggested, for example, in Greaves (2013). On a simple way of spelling out this idea, the ultimate epistemic value is accuracy: closeness of an agent's beliefs to the truth.
Accuracy evidently pertains to agent stages. Your belief state tomorrow might be more accurate than your belief state today. It's not obvious, however, whether your future accuracy matters to what you should believe today. The accuracy of other people, for example, seems clearly irrelevant. We don't want to say that you should adopt certain beliefs today because this would cause other people to become more accurate.
But what about your own future accuracy? A well-known argument for conditionalization, due to Greaves and Wallace (2006), suggests that it matters: Conditionalization maximizes expected future accuracy. Why is this relevant? Because your future accuracy matters. Epistemically speaking, you should adopt a plan to update your beliefs in a way that is conducive to your future accuracy.
This is, of course, controversial, but let's run with it. The argument for conditionalization looks only one step ahead. But if the accuracy of you next stage matters, then surely the accuracy of the stage after that can't be irrelevant. We need to aggregate the accuracy of all your future stages.
But then we run again into the problems of aggregation.
If we evaluate a plan to update your current beliefs by the average accuracy of your future beliefs, we get very strange recommendations.
Imagine a fair coin will be flipped tonight: heads you will die soon after waking up; tails you will live a long life. By waking up confident that the coin has landed heads, you can strongly improve the average accuracy of your future beliefs in case of heads, at comparatively little cost to the average accuracy of your future beliefs in case of tails. For note that if the coin lands tails, you're going to find this out very soon – from the fact that you're still alive. So if you wake up believing that the coin has landed heads, your erroneous belief will soon be corrected, and its effect on your average accuracy across your long life will be small. So, if we evaluate your update plan by the expected average accuracy of your future beliefs, we have to say that you should plan to wake up confident that the coin has landed heads. This seems clearly wrong.
What if we evaluate your update plan by the expected total accuracy of your future beliefs? Take the same coin flip scenario, with the added stipulation that if the coin lands tails, you will be made to forget that you would have died if it had landed heads. That is, if the coin lands tails, you won't learn about this outcome. Let's assume that you will simply retain the credence (about the outcome of the coin flip) with which you wake up for the rest of your long life. The total accuracy measure then implies that you should wake up confident that the coin has landed tails.
This may not be obvious. To see the point, let's pretend that the only relevant question is the outcome of the coin flip, so that the inaccuracy of a credence function Cr can be measured by (say) the squared difference between Cr(Heads) and the truth-value of Heads. Let x be the credence you assign to Heads when you wake up. If the coin lands heads, the total inaccuracy of your future belief states is \( (1-x)^{2} \). (There is only one belief state before you die.) If it lands tails, the total future inaccuracy is \( x^{2} + x^{2} + x^{2} ... \). (Many belief states, and you learn nothing new about the coin flip.) Both outcomes are equally probable. To minimize the expected total inaccuracy (= maximize the expected total accuracy) of your future beliefs, you should wake up with a low value of x. This also seems clearly wrong.
In the practical and the moral domain, we could avoid the problems of aggregation by simply rejecting the relevant separability assumption. This doesn't seem to be an option in the epistemic domain.